Improve Dashboard performance for large Octopus installations

[MarkSiedle]

For large installations (with hundreds of users logging into Octopus throughout the day), we noticed the dashboard queries causing systemic performance issues due to the impact on the database.

When the combination of projects and environment gets into the high 100's, performance starts becoming pretty bad. There are several levels contributing to this:

• The Dashboard is polled by all open Octopus UI instances even if they are not on focused browser tabs
• The Dashboard view is slow due to how it scans the ServerTask table
• All the data is serialised to the client, the only filtering being done in server memory is the "Configure Dashboard" options. The text search and top X filter is done client side
• The Previous Successful dashboard data is only used internal
• The Dashboard view is also used by other code paths (eg create release)

Potential sources

The UI polls the Dashboard API whenever a user has a browser tab on the Dashboard. This could even be an unfocused browser tab. Another common use case is to open Octopus, deploy what you need, then create a new tab and keep on working (without realising that the unfocused Octopus window(s) may still be polling the API).

Multiply this across a large installation and you've got a ready-made DDoS attack against your own Octopus Server.

We do some caching of the dashboard on the Octopus Server, but empirical data from some of our customers shows we need to improve have the dashboard scales across large installations.

Potential Solutions

A problem this complex will likely have lots of small enhancements to fix the underlying problems.

Optimize queries

Yep! Here are some suggestions from Azure SQL DB: #2767

Stop polling the dashboard if the browser tab is not visible

Could we consider disabling the dashboard timers when windows are unfocused (re-enable when window becomes focused)?

Consider auto-sign out

Or look at an auto-signout configuration feature that large installations could opt-in to. Eg. After X mins of inactivity, sign out.

If we "do" look at auto-sign-out, make sure we consider that this does not conflict with the new auto-signon functionality coming out in 3.5 when you have a single auth-provider enabled ...otherwise the auto-signout will trigger and we'll just auto-sign them in again :)

Improved cache busting

We could consider using the Events table to determine if the cached dashboard data in the Octopus Server node's in-memory cache is actually out of date. If there haven't been any deployment-related events since the dashboard was last queried, return the cached version.

Getting involved

If you have some ideas from your own experience in working with the Octopus Dashboard, like query optimisations based on your SQL Server statistics, we would love to hear about them!

[vanessalove]

Dashboard performance hit with unfocused-windows

[MJRichardson]

With the attached closed issues, we feel we have done what we can short of a substantial re-design of the dashboard.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you think you've found a related issue, please contact our support team so we can triage your issue, and make sure it's handled appropriately.