The Pipelines project targets two user groups. One of them are new users to Python, who might be familiar with basic SQL but don't know programming. We'll call them new users It's critical to the mission of Pipelines that it's accessible to new users.
New users should be able to easily:
Use pipelines to perform ETL tasks
Take advantage of advanced pipelines features like data orchestration (Prefect integration as an example) "out of the box"
New users do not need to be able to perform more advanced tasks like:
Writing new pipes
Extending the pipelines framework with new functionality, like adding data orchestration plugins.
To focus the conversation, below is a copy of the current example script in the pipelines branch (as of 2/27/2024). I believe it captures the surface area of functionality new users will be expected to engage with.
The Pipelines project targets two user groups. One of them are new users to Python, who might be familiar with basic SQL but don't know programming. We'll call them new users It's critical to the mission of Pipelines that it's accessible to new users.
New users should be able to easily:
New users do not need to be able to perform more advanced tasks like:
To focus the conversation, below is a copy of the current example script in the
pipelines
branch (as of 2/27/2024). I believe it captures the surface area of functionality new users will be expected to engage with.