OctopusDeploy / Issues

| Public | Bug reports and known issues for Octopus Deploy and all related tools
https://octopus.com
162 stars 20 forks source link

Improve Dashboard performance for large Octopus installations #2850

Closed MarkSiedle closed 7 years ago

MarkSiedle commented 8 years ago

For large installations (with hundreds of users logging into Octopus throughout the day), we noticed the dashboard queries causing systemic performance issues due to the impact on the database.

When the combination of projects and environment gets into the high 100's, performance starts becoming pretty bad. There are several levels contributing to this:

Potential sources

The UI polls the Dashboard API whenever a user has a browser tab on the Dashboard. This could even be an unfocused browser tab. Another common use case is to open Octopus, deploy what you need, then create a new tab and keep on working (without realising that the unfocused Octopus window(s) may still be polling the API).

Multiply this across a large installation and you've got a ready-made DDoS attack against your own Octopus Server.

We do some caching of the dashboard on the Octopus Server, but empirical data from some of our customers shows we need to improve have the dashboard scales across large installations.

Potential Solutions

A problem this complex will likely have lots of small enhancements to fix the underlying problems.

Optimize queries

Yep! Here are some suggestions from Azure SQL DB: #2767

Stop polling the dashboard if the browser tab is not visible

Could we consider disabling the dashboard timers when windows are unfocused (re-enable when window becomes focused)?

Consider auto-sign out

Or look at an auto-signout configuration feature that large installations could opt-in to. Eg. After X mins of inactivity, sign out.

If we "do" look at auto-sign-out, make sure we consider that this does not conflict with the new auto-signon functionality coming out in 3.5 when you have a single auth-provider enabled ...otherwise the auto-signout will trigger and we'll just auto-sign them in again :)

Improved cache busting

We could consider using the Events table to determine if the cached dashboard data in the Octopus Server node's in-memory cache is actually out of date. If there haven't been any deployment-related events since the dashboard was last queried, return the cached version.

Getting involved

If you have some ideas from your own experience in working with the Octopus Dashboard, like query optimisations based on your SQL Server statistics, we would love to hear about them!

vanessalove commented 7 years ago

Dashboard performance hit with unfocused-windows

MJRichardson commented 7 years ago

With the attached closed issues, we feel we have done what we can short of a substantial re-design of the dashboard.

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. If you think you've found a related issue, please contact our support team so we can triage your issue, and make sure it's handled appropriately.