Closed amindadgar closed 3 months ago
[!WARNING]
Review failed
The pull request is closed.
The recent updates primarily focus on enhancing the scheduling behavior of various DAGs by incorporating the max_active_runs=1
parameter across multiple files, ensuring only one active run at a time. Additionally, the catchup=False
parameter is added to several DAG declarations to prevent task backlog. These changes aim to streamline task execution and improve overall system stability.
Files | Change Summaries |
---|---|
dags/github.py |
Added max_active_runs=1 to the DAG declaration. |
dags/hivemind_discord_etl.py |
Added max_active_runs=1 and removed catchup=False in DAG declarations for hivemind_discord_etl and start_discord_vectorstore . |
dags/hivemind_discourse_etl.py |
Added catchup=False and max_active_runs=1 to DAG configurations for discourse_vector_store and discourse_summary_vector_store . |
dags/hivemind_github_etl.py |
Added catchup=False and max_active_runs=1 to the DAG definition. |
dags/hivemind_google_drive_etl.py |
Added catchup=False and max_active_runs=1 to the DAG definition for gdrive_vector_store . |
dags/hivemind_mediawiki_etl.py |
Added catchup=False and max_active_runs=1 to the DAG declaration. |
dags/hivemind_notion_etl.py |
Added catchup=False and max_active_runs=1 to the DAG declaration. |
In the land of data's endless run,
A rabbit hopped, the tasks begun,
Withmax_active_runs
set to one,
Andcatchup
gone, the race was won.
Now DAGs do dance in perfect tune,
Under the bright scheduling moon. 🌕
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
Summary by CodeRabbit