search

Cargill / pipewrench

Data pipeline automation tool
Apache License 2.0
25 stars 36 forks source link

airflow support for pipewrench #28

Open SandishKumarHN opened 6 years ago

SandishKumarHN commented 6 years ago

airflow support for pipewrench

afoerster commented 6 years ago

@SandishKumarHN this will be a brand new set of templates?

SandishKumarHN commented 6 years ago

Yes, in airflow task-id would be like this example-> sqoop_createexec{{ conf.sourcedatabase.name }}{{ table.source.name }} and filenames would be same but .py or we can do like examples-> create_execute_sqoop_parquet_hdfs_impala.py and clean_sqoop_parquet_hdfs_impala.py

afoerster commented 6 years ago

I'm not sure I'm following, share a WIP branch when you get a rough outline.