FredHutch / motuz

Motuz - A web based infrastructure for large scale data movements between on-premise and cloud
MIT License
104 stars 12 forks source link

start celery pool with concurrency == number of cpus * 3 #410

Open dtenenba opened 2 years ago

dtenenba commented 2 years ago

Since our jobs are IO-bound they are not CPU intensive and we can make the pool deeper.

For reference, pool size can be increased at runtime by docker exec'ing into the celery container and running:

source ./load-secrets.sh
celery -A api.tasks control pool_grow 20 # adds 20 to current size