Open JamesKunstle opened 9 months ago
Should have Flower running so that we can see which task are running when the system gets stuck like this.
This happens when the backend monitoring task (that runs the Data Ready callback) runs ahead of other callbacks that collect data- it won't finish because the callbacks that collect data won't run.
This will be fixed when clientside query worker polling is enabled.
The task that triggers the "data loading" animation to be rendered wait on all registered query worker tasks to complete. This task simply polls all of the query worker tasks to see if they're finished, and sleeps in between checks.
If many of these monitor tasks are being processed by Celery worker processes they can actually block further querying process. They're waiting for data to be queried from Augur, but they're blocking workers from continuing to do this work.
More details below.