lisad / phaser

The missing layer for complex data batch integration pipelines
MIT License
4 stars 1 forks source link

Allow variable values to be passed in on command line #153

Open lisad opened 3 months ago

lisad commented 3 months ago

Issue #152 suggested this feature idea

When working with data from different sources, different days, different URLs etc, the pipeline should be able to be launched with variable names

E.g: data team periodically downloads "employees.csv" from multiple companies and it's also important what day it's downloaded

> python3 -m phaser run employee_pipeline output "sources/employees.csv" --var company_name="Example" --var date="2024-06-06"

I guess this would be done like this:

jeffkole commented 2 months ago

Update phaser-example to use this new feature