As part of debugging for this PR, I also manually deleted about 4 GB of logs from the staging server. This should give us enough room to cover us while we review and deploy this change.
Testing instructions
Start the LA Metro server in the background: cd la-metro-councilmatic && docker-compose up -d app
Start the dashboard server: cd ../la-metro-dashboard && docker-compose up
Navigate to http://localhost:8080 and turn ON both airflow_log_cleanup and airflow_db_cleanup
After airflow_log_cleanup and airflow_db_cleanup have run, check their log output to confirm that they deleted logs and database tables older than 30 days
If you don't have logs or database tables older than 30 days, you can also manually trigger these DAGs and set the config attribute maxDBEntryAgeInDays or maxLogAgeInDays to make the window smaller (e.g. a complete config object might look like {"maxDBEntryAgeInDays": 7} )
Overview
Following the advice in https://blog.clairvoyantsoft.com/automated-maintenance-for-apache-airflow-8d844f32737d, this PR defines two meta-DAGs,
airflow_db_cleanupandairflow_log_cleanup, that run daily and clean up old logs and database entries for the app.Notes
As part of debugging for this PR, I also manually deleted about 4 GB of logs from the staging server. This should give us enough room to cover us while we review and deploy this change.
Testing instructions
cd la-metro-councilmatic && docker-compose up -d appcd ../la-metro-dashboard && docker-compose upONbothairflow_log_cleanupandairflow_db_cleanupairflow_log_cleanupandairflow_db_cleanuphave run, check their log output to confirm that they deleted logs and database tables older than 30 daysmaxDBEntryAgeInDaysormaxLogAgeInDaysto make the window smaller (e.g. a complete config object might look like{"maxDBEntryAgeInDays": 7})