Upgrade: Add maintenance DAGs

Metro-Records / la-metro-dashboard

An Airflow-based dashboard for LA Metro

4 stars 0 forks source link

Closed hancush closed 11 months ago

hancush commented 1 year ago

Add maintenance DAGs to run airflow clean and clean up log files.

antidipyramid commented 1 year ago

@hancush Is it reasonable to delete metadata and log files older than 30 days?

hancush commented 1 year ago

@antidipyramid Very!

antidipyramid commented 1 year ago

@hancush Also, should we be doing some sort of backup before running the airflow db clean command?

hancush commented 1 year ago

@antidipyramid The logs are pushed to S3 automatically. 🙃