sni
commented
3 years ago Interesting idea. But i am really wondering why the refresh takes up to 10min. I've seen big sites that take up to 1minute. But 10 minutes really sounds like something is wrong (unless this is a super small connection). And if its a low bandwidth connection, then passing queries through would make the initial sync even slower. LMD show a detailed list how long each table takes to sync (when debug log is enabled). Is there anything suspicious? Maybe synchronization after a reload could be improved by not loading everything and do a smarter sync. Fetching everything was just the easiest option and so far it was sufficient for all sites.
steffen-poulsen
We have two backend types that are unfortunately not a hand-in-glove fit for the current implementation of LMD.
One type is our Icinga backends, which appear to be bad at signaling that they have fresh data for the LMD. This means that they will only get refreshed every 5 minutes, even if they have new data to offer every second.
Another type is our huge Nagios backends (100.000s of checks), that can take up to 10 minutes or more for the LMD to refresh. These Nagios backends do have the advantage over the Icinga backends, that once they have been refreshed, the delta update works fast.
Unfortunately, we have a very busy environment with a lot of changes coming in. Therefore we would like to reload the backends often, for the configuration updates to reflect as quickly as possible. The Nagios reload in itself does not present a problem for this to happen, as it reloads fast, typically in 30s. But, the 10m LMD refresh is a challenge, as we would like to reload more often than the 30m that we feel is possible for now - a reload every 30m leaves 20m of responsive GUI time with the Nagios backends.
This 10m delay time while waiting for the LMD to refresh the backend means that we are often left with stale check results in our console. And worse, with no visibility of acknowledgements and comments that we have added. The check results will be updated eventually, and acknowledgements and comments will show up eventually (after 10m), so the LMD functionality is ok. Just, delayed.
We love the LMD very much, and we would like to investigate options for bringing interactions and responses more "online", even with these two badly behaving backend types that we have.
An idea that we would like to propose, is to add an option to pass requests directly through to the backend, if the backend is online, but its cache is not fresh.
The LMD cache would still be refreshing in the background as it currently does. And whenever the cache is fresh, the LMD will use the cache as its source (putting less CPU load on our Nagios backends). And, whenever the LMD cache is not fresh but the backend is up, the LMD will use the backend as source (making our Icinga backends responsive to work with, even if the LMD cache would seldom be used).
I was playing around with a sequence diagram, I am attaching it just to show my current understanding of how things currently work with the LMD.
We don't know if this kind of change is possible at all, but we would love to hear your thoughts.