The airflow scheduler currently outputs over 200Mb of logs every day, and the majority of them are for DAGs that haven't run. It would be good to reduce this.
I could manage this by automating deletion of logs, but I'd rather explore configuring airflow to output fewer logs first.
To achieve this, I'll want to add the airflow.cfg file to version control, which will involve mounting the file to the host and confirming that no sensitive information would be leaked.
The airflow scheduler currently outputs over 200Mb of logs every day, and the majority of them are for DAGs that haven't run. It would be good to reduce this.
I could manage this by automating deletion of logs, but I'd rather explore configuring airflow to output fewer logs first.
To achieve this, I'll want to add the airflow.cfg file to version control, which will involve mounting the file to the host and confirming that no sensitive information would be leaked.