search

gsemet / kube-airflow

Helm chart to run production Airflow/Celery on Kubernetes
20 stars 8 forks source link

Workers vs DAG concurrency #9

Closed minh5 closed 6 years ago

minh5 commented 6 years ago

Hey @Stibbons I've been tinkering around with your repo and one thing I noticed is that somehow a worker will fail at all task while another one will succeed. As I'm tuning Airflow, I'm curious to see what would be the tradeoff between more worker statefulsets vs increasing the amount of dag_concurrency within the Airflow configuration?

gsemet commented 6 years ago

You'll welcome to add you information here. We use celery (not airflow) with a celery concurrency of 4 or 8 depending on how many CPU we initialize the spec with.

For this airflow chart, the simpler would be to let user set the dag_concurrency

minh5 commented 6 years ago

Thanks I just wanted to make sure if there are any preferred way of scaling. Closing