slominskir
commented
6 months ago Worth mentioning that a global dynamic filter rule means ALL records must be re-processed on update. If an operator decides to experiment with global filters each update (try adding a new Location to the global filter set) results in every notification being invalidated and re-issued. At the moment that is 25,483 notifications. Messages are pretty small so probably not a huge deal, but worth mentioning this doesn't scale particularly well.
This has implications for Kafka compacted topics as well. We'll need to carefully tune the notifications topic to ensure old messages are somewhat aggressively compacted, but this is a balancing act with allowing temporarily disconnected clients a chance to catch back up. For notifications probably makes sense to ensure very aggressive compaction and rely on convention that clients always re-wind and start from beginning on re-connect. Probably need to disable Kafka broker-side offset tracking completely.
Currently the
FilteredOverride is applied manually and individually and explicitly to alarms (one-by-one keyed on name with 100 max SHIFT click selectable at once). This is how Shelved and Disabled work as well. However, the current limit of 100 alarms displayed / selectable on the admin GUI at any given time means filtering out thousands of alarms all at once is painful. For Shelved and Disable the current system is probably fine as the number of shelved and disabled alarms is likely small. However, it appears alarms that need to be filtered out due to machine program may number in the thousands. In extreme cases we may need to silence the majority of alarms during a commissioning period isolated to the Injector for example. It turns out that during SAD/Off periods about 14,000 alarms may be active. An alternative would be to ask alarm producers to design their alarms to NOT be actively alarming when the machine is OFF in their location.We either need to provide a way to easily update overrides by name in bulk (thousands of alarms at once) or perhaps define a dynamic filter system that does so dynamically and continuously based off non-unique attributes. For example the effective-alarm-process could reads a "filter rules" topic and apply them continuously as registration changes are made. This assumes that filtering usually occurs based on a non-unique attribute such as location or name pattern. We actually have a very early deprecated prototype of a dynamic processor here: https://github.com/JeffersonLab/alarms-filter