search

OSC / ondemand

Supercomputing. Seamlessly. Open, Interactive HPC Via the Web
https://openondemand.org/
MIT License
288 stars 106 forks source link

Dynamic interval for batch_connect/sessions.js polling #3729

Open CSC-swesters opened 2 months ago

CSC-swesters commented 2 months ago

The batch_connect/sessions.js polling interval is currently set to a static value via POLL_DELAY (link to the code), and defaults to 30_000 ms. We recently observed that the amount of httpd worker threads was running out due to a high number of users, and perhaps other causes.

Either way, at this time, we noticed that the amount of sessions.js traffic was quite significant. Analyzing our logs, I can see that around 10% of our HTTP requests consist of GET /pun/sys/dashboard/batch_connect/sessions.js?... requests.

A big factor in this is probably that users open the compute job in a new browser tab, and then the tab which is showing the app card keeps polling the sessions.js endpoint. This is quite unnecessary, since users presumably aren't looking at the page.

I would not want to set the POLL_DELAY too high either, because it will affect responsiveness while users are waiting for the job to get scheduled and started.

Would it be possible to dynamically adjust the POLL_DELAY, such that it increases substantially when the job goes into a "Running" state?

For example, I could see it being useful to poll every 10 seconds while the job is queueing and starting, but then changing to 300 second polling intervals for the clients that are only looking at "Running" jobs (with a green banner).

I also notice that polling happens for cancelled or stopped jobs. This is a bit unnecessary as well, and leads to unnecessary traffic. Maybe a Refresh button could be used instead, or let the user refresh the browser page?

Let me know what you think!

robinkar commented 2 months ago

Also worth noting that if the user has multiple tabs open with the interactive sessions page, each of the tabs will get the status of the sessions every POLL_DELAY ms. Would it make sense to have the data cached in Rails for a certain amount of time to serve the same data regardless of which tab queries it?

johrstrom commented 2 months ago

Yes likely. We can cache html partials, though we don't as of yet.

johrstrom commented 1 week ago

I can't get this into 4.0 with the timeline as it is. As one positive, we no longer query forever. The javascript will actually stop making requests when it encounters a 401 as I've seen that happen a lot - folks have tabs open and they're not authenticated anymore but continue to make queries.