search

sul-dlss / libsys-airflow

Airflow DAGS for migrating and managing ILS data into FOLIO along with other LibSys workflows
Apache License 2.0
5 stars 0 forks source link

Configure priority weight for polling dag #1370

Closed shelleydoljack closed 3 weeks ago

shelleydoljack commented 3 weeks ago

When running the dags in airflow-stage, it looks like all the running slots in the pool get taken up by the polling dag. This screenshot shows 14 running task for the polling dag and 16 tasks in the digital_bookplate_instances DAG queued. Screenshot 2024-10-28 at 1 56 07 PM

The priority weight of the poling dag tasks are 3 while the ones for digital_bookplate_instances are queued with priority weight 5. I thought higher number means more priority. 🤷 https://airflow.apache.org/docs/apache-airflow/2.8.3/administration-and-deployment/priority-weight.html#priority-weights

jgreben commented 3 weeks ago

I think because the weighting os on a per-task basis, it it probably scoped to its own dag and not across dags, even though the polling dag is triggered by the D-B-instances dag.