Include file_dependencies as variable for Copy Commands to extend for cases where copy shall be skipped if sql_files is not changed.

mara / mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

MIT License

2.08k stars 102 forks source link

Include file_dependencies as variable for Copy Commands to extend for cases where copy shall be skipped if sql_files is not changed. #17

Closed jieyinybl closed 5 years ago

jieyinybl commented 5 years ago

We do not have yet file_dependencies as variable for the Copy command. I recently had a case where I only need to copy data into a table for the initial load. Hence I think including the file_dependencies as variable for the copy command could handle cases in ETL pipeline, where the copy command shall be skipped if the sql_files stay the same.