search

craftcms / cms

Build bespoke content experiences with Craft.
https://craftcms.com
Other
3.26k stars 632 forks source link

Stampede of requests to `queue/get-job-info` #10632

Closed nickdunn closed 2 years ago

nickdunn commented 2 years ago

Description

I'm debugging a server event which brought down a site.

In the logs I'm seeing a large number of requests matching POST /index.php?p=cms/actions/queue/get-job-info&limit=50&dontExtendSession=1, which is the UI checking the status of the queue. I seem to get a stampede of requests in quick succession over the course of a couple of minutes.

e.g.

image

When a save event occurred and jobs were added to the queue, the AJAX requests are fired to monitor progress. This would yield maybe 3–5 AJAX requests in the current tab, one every 500ms or so until the queue is complete a few seconds later.

My further suspicion is the CMS user may have had many dozens of Craft entries open (potentially more, who knows, trying to find this in the logs too!) and that all of these tabs would have then fired AJAX requests too, checkin the queue, even though they're not the active browser tab.

My suspicion is backed up with the fact that I'm also seeing a number of GET /index.php?p=cms/actions/users/session-info&dontExtendSession=1 throughout this stampede of requests, suggesting maybe they've had a load of tabs open for some time.

  1. Is it possible to configure the polling frequency of the endpoint, say to once per second, to reduce the number of requests?
  2. Is it possible that having a lot of tabs open would have caused a stampede of requests? Sometimes the queue gets stuck and the sidebar shows "Failed", and I don't know if that means page views are always issuing more requests to check the queue status.
  3. If 2 above is a possibility, could/should https://developer.mozilla.org/en-US/docs/Web/API/Document/visibilityState be used to prevent a stampede?
  4. Is there anything else that could be issuing a large number of these internal requests?

Additional info

brandonkelly commented 2 years ago

I wasn’t aware of document.visibilityState! Looks like we should be able to avoid these polling requests with that.

nickdunn commented 2 years ago

Thanks Brandon.

Can you confirm my assumption... that if you have multiple tabs open, if a queue job is started, then all open tabs will start polling?

Does this also happen if the sidebar shows the "Failed" queue state, will they still start polling?

mattgrayisok commented 2 years ago

Just adding my support for this. We just had a site fall over due to too many CP tabs open causing a constant 10req/s hitting the get-job-info endpoint and maxing out DB CPU (I think it maybe runs an order by on the queue which soaks up a lot of CPU if there's a large number of pending jobs? I haven't checked very closely)

Was also going to suggest visibilityState which also has events that'll hopefully be useful in this context:

https://developer.mozilla.org/en-US/docs/Web/API/Document/visibilitychange_event

brandonkelly commented 2 years ago

I had hoped to get this into 4.0, but couldn’t get to it in time. It’s done for 4.1 now though!

If you want to test, you can change your craftcms/cms requirement in composer.json to 4.1.x-dev, and run composer update.

brandonkelly commented 2 years ago

Craft 4.1 is out with this change ✨