celery / billiard

Multiprocessing Pool Extensions
Other
420 stars 251 forks source link

BF: restrict value of get_fdmax #417

Open yarikoptic opened 6 days ago

yarikoptic commented 6 days ago

In my case I kept finding celery running at 100% and doing nothing. py-spy pointed to the close_open_fds and then ulimit inside the container showed gory detail of

❯ docker run -it --rm --entrypoint bash dandiarchive/dandiarchive-api -c "ulimit -n"
1073741816

situation is not unique to me. See more at

I verified that with this fix my celery container gets unstuck and proceeds to report useful errors ;)