dataverbinders / nl-open-data

A Flexible Python ETL toolkit for datawarehousing framework based on Dask, Prefect and the pydata stack
https://dkapitan.github.io/nl-open-data
MIT License
0 stars 1 forks source link

Remove temp dir if `main` fails in `statline_bq` flow #89

Closed galamit86 closed 3 years ago

galamit86 commented 3 years ago

Currently, within statline_bq flow, remove_dir() which runs as a clean up task in most flows, receives its input parameters from main(). However, if main() fails, no input is given to remove_dir(), and it fails as well.

remove_dir() should alway run and its input must be independent from the state of main().