search

ControlSystemStudio / phoebus

A framework and set of tools to monitor and operate large scale control systems, such as the ones in the accelerator community.
http://phoebus.org/
Eclipse Public License 1.0
90 stars 90 forks source link

Alarm Tree performance #2411

Closed georgweiss closed 2 years ago

georgweiss commented 2 years ago

I have a faint recollection that this has been discussed in some context, but my users are complaining about bad performance of Phoebus when used over remote desktop (NoMachine).

I am able to kind of trigger bad UI response if I start the Alarm Tree application. The default configuration is a "simple" tree with one top level node containing a large number of PVs, many of which are disconnected. When this configuration is loaded into the tree view, the error log looks like below. Based on the time stamp of these messages it seems the total time to load the configuration is almost 5 min.

Clues, anyone? 

2022-10-11 09:45:29 INFO [org.phoebus.applications.alarm] Reading from start of ACCP
2022-10-11 09:45:29 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:34 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:34 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:35 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:35 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:43 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:43 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:43 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:44 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:44 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:45 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:45 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:46 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:46 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:47 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:47 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:51 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:51 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:51 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:52 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:52 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:55 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:55 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:55 INFO [org.phoebus.applications.alarm] Alarm tree changes start
2022-10-11 09:45:56 INFO [org.phoebus.applications.alarm] Alarm tree changes end
2022-10-11 09:45:56 INFO [org.phoebus.applications.alarm] Alarm tree changes start
.
.
.
.
kasemir commented 2 years ago

Have you tried after https://github.com/ControlSystemStudio/phoebus/pull/2389? For me, that reduced load times from about 20 seconds to 2.

kasemir commented 2 years ago

.. with the newly introduced alarm_tree_startup_ms setting at 2000 ms. You can try larger if you have longer delays.

georgweiss commented 2 years ago

Thanks @kasemir, that made all the difference...

As it happens, I made a release on the commit just before #2389...

kasemir commented 2 years ago

Do you have some data points for your NoMachine issue:

I was looking at a setup with about 1000 PVs via 'Thinlinc', and load time went from 20 sec to 2 sec. Those 2 sec are basically the alarm_tree_startup_ms default setting of 2000 ms to wait 2 sec, and then it renders very quickly. Waiting only 1 sec resulted in then many rendering steps as more updates come in, giving overall a much longer startup.

Note that an operational change of the alarm tree status, like running the example from https://github.com/ControlSystemStudio/phoebus/blob/master/app/alarm/model/src/test/java/org/phoebus/applications/alarm/AlarmConfigProducerDemo.java with 10k PVs where about 1000 of them change every few seconds tends to be handled fine. What bites is a change in the tree layout, adding or removing tree nodes, so there the initial delay to get the complete tree and render it once helped a lot.

georgweiss commented 2 years ago

Number of nodes: 3212 (all in one top-level component node) Value for alarm_tree_startup_ms: default (2000) Original load time: anything between 3 - 5 min Improved load time: ~2s

kasemir commented 2 years ago

Really neat that the default startup ms works there as well. Reducing 3+ minutes to 2 seconds qualifies as a-ma-zing.

georgweiss commented 2 years ago

Amazing indeed. Qualifies for a beer.