oss-aspen / 8Knot

Dash app in development to serve open source community visualizations using GitHub data from Augur. Hosted app: https://eightknot.osci.io
MIT License
46 stars 59 forks source link

BUG: worker status task (checks status of all queries) causes deadlock if many of its kind are running. #525

Open JamesKunstle opened 9 months ago

JamesKunstle commented 9 months ago

The task that triggers the "data loading" animation to be rendered wait on all registered query worker tasks to complete. This task simply polls all of the query worker tasks to see if they're finished, and sleeps in between checks.

If many of these monitor tasks are being processed by Celery worker processes they can actually block further querying process. They're waiting for data to be queried from Augur, but they're blocking workers from continuing to do this work.

More details below.

JamesKunstle commented 9 months ago

Should have Flower running so that we can see which task are running when the system gets stuck like this.

JamesKunstle commented 6 months ago

This happens when the backend monitoring task (that runs the Data Ready callback) runs ahead of other callbacks that collect data- it won't finish because the callbacks that collect data won't run.

This will be fixed when clientside query worker polling is enabled.