oss-aspen / 8Knot

Dash app in development to serve open source community visualizations using GitHub data from Augur. Hosted app: https://eightknot.osci.io
MIT License
47 stars 59 forks source link

Investigate Celery worker configuration to make sure we're using best practices #379

Open JamesKunstle opened 1 year ago

JamesKunstle commented 1 year ago

Here's an article that summarizes this configuration: https://progressstory.com/tech/python/production-ready-celery-configuration/

This should let us remove 'job.forget()' code in index_callbacks.py because we can configure the workers to ignore the results from the process that they run.

Should also set the prefetch multiplier so that if 'n' requests are made and 'n' workers are available, all of the requests start at once rather than one of the workers prefetching 'm<n' of the requests, preventing them from starting immediately.