Icinga / icinga2

The core of our monitoring platform with a powerful configuration language and REST API.
https://icinga.com/docs/icinga2/latest
GNU General Public License v2.0
2k stars 574 forks source link

Update a child's reachable state as soon as possible #10143

Open nilmerg opened 1 month ago

nilmerg commented 1 month ago

Given a three level dependency hierarchy, the child on the lowest level is not unreachable in case a parent on the first level goes down, unless a check result arrives afterwards.

graph LR;
    ChildHost-->pa["ParentHostA (Group 1)"];
    ChildHost-->pb["ParentHostB (Group 1)"];
    pa-->ga["GrandParentA (Group 2)"];
    pa-->gb["GrandParentB (Group 2)"];
    pb-->ga["GrandParentA (Group 2)"];
    pb-->gb["GrandParentB (Group 2)"];

Here, ChildHost must be unreachable once both, GrandParentA and GrandParentB, are down. Sooner or later.

Expected behavior

I'm unsure whether we should update every child's reachable state in such a case. Since this will be updated anyway, once a check is performed, it is highly dependent on the interval of the check:

If the object is currently UP/OK, there's no real need to update it, since everything "is fine" and no-one should worry about it. Though, what if the object already has a problem? In case the check interval is relatively high, the reachable state is not going to update soon enough. But of course, the reason why it has already a problem, might not necessarily be related to a parent. So, maybe update such an object only if it's state is UNKNOWN?

Another very different case though, is in case the dependency configuration mandates that checks get disabled once the parent goes down. Then the child will never be checked again and the reachable state won't be updated at all without an explicit check issued by a user. Here I expect that, before disabling checks, the reachable state must be updated as well.

--

So, I'm certain what to expect in cases where checks will be disabled. But if that's not the case, is it really required to traverse the entire hierarchy to every child related in some way to the parent in question?