oss-know / airflow-jobs

Apache License 2.0
6 stars 19 forks source link

Set different job intervals for daily sync DAGs #184

Closed crystaldust closed 1 year ago

crystaldust commented 1 year ago

Daily sync DAGs needs to be scheduled by different intervals to balance resource consuming. The policy is to 2 level intervals in variables with different priority:

high: daily_sync_{DAG_NAME}_interval
middle: daily_sync_interval

If the interval variables above is not configured, do not trigger the DAG periodically.