TogetherCrew / airflow-dags

1 stars 1 forks source link

[hivemind] avoid running multiple instances of a DAG in parallel! #179

Closed amindadgar closed 3 months ago

amindadgar commented 3 months ago

For the case of hivemind, we're always getting the documents and processing them. In case of running a DAG tasks in parallel, the data could be mixed up and for a single document, we might end up having duplicate or no nodes for it. We should ensure that a DAG won't be run in parallel.