search

button-inc / emissions-elt-demo

2 stars 0 forks source link

refactor: DAG refinement #202

Open joshgamache opened 1 year ago

joshgamache commented 1 year ago 
### Tasks
- [ ] A few of the current DAGs have a run-order dependence. It would be good practice to combine these into a single DAG for flow handling 
- [ ] DAGs, as they are, currently only load initial data but not subsequent data. A separate _initial load_ DAG should exists, which is then supported by triggered/scheduled DAGs to ingest new data.

- [ ] DAGs are triggered manually. A DAG should be created to monitor the storage dropbox and send to the correct pipeline.