eschoeller
commented
3 years ago So I went ahead and blew away my retention.dat files on all three naemon machines and then truncated the report_data and notification tables to try and 'start over'. Things started to work in a more synchronized manner. Then I shut off merlin and naemon on one of the machines for about 12 hours. As I would expect the report_data table on the machine that wasn't running naemon became out of date. I then turned merlin and naemon back on. That machine started to 'catch up' and I watched the report_data table start to become more identical to the two other machines. While this happened the naemon log on the machine that was not running was flooded with SERVICE ALERT messages for each 'update'. But in the process of this unfortunately that machine started to blast out notifications for issues that had already been reported on earlier in the day. But only a certain subset of notifications (we had a very busy day of failures today). Now as it stands all three seem to be synchronized back together, albeit just off by a couple IDs in the report_data table:
| 11657 | 1625622927 | pdu-f10 | Infeed Power Factor | 1 | 0 | 84 |
| 11663 | 1625623228 | pdu-f10 | Infeed Power Factor | 1 | 0 | 85 |
| 11679 | 1625623834 | pdu-f10 | Infeed Power Factor | 1 | 0 | 86 |
+-------+------------+---------------------+---------------------+-------+------+-------+
375 rows in set (0.015 sec)
| 11657 | 1625622927 | pdu-f10 | Infeed Power Factor | 1 | 0 | 84 |
| 11663 | 1625623228 | pdu-f10 | Infeed Power Factor | 1 | 0 | 85 |
| 11679 | 1625623834 | pdu-f10 | Infeed Power Factor | 1 | 0 | 86 |
+-------+------------+---------------------+---------------------+-------+------+-------+
375 rows in set (0.015 sec)
| 11661 | 1625622927 | pdu-f10 | Infeed Power Factor | 1 | 0 | 84 |
| 11664 | 1625623228 | pdu-f10 | Infeed Power Factor | 1 | 0 | 85 |
| 11675 | 1625623834 | pdu-f10 | Infeed Power Factor | 1 | 0 | 86 |
+-------+------------+---------------------+---------------------+-------+------+-------+
375 rows in set (0.013 sec)Do I have a configuration error somewhere? I was potentially thinking that having check_freshness enabled could be messing up merlin? or is that necessary for making the redundancy work?
jacobbaungard
My apologies for posting this here - I was looking for the op5 mailing list (which I was subscribed to years ago) but it appears op5.org is gone ...
I am switching from a very old version of Nagios/Merlin ... probably 6 years old at least. I am seeing some behavior with the newer version that I do not understand. I am running Naemon 1.2.4-1 and pulled Merlin from git today.
In the past, the nagios node running a check would log the 'SERVICE ALERT' for that check. I had all three of our peered nagios/merlin machines syslogging to each other so I had a nice view of where checks were being run and what was happening.
With the new version it appears certain events are being "echoed" by all the naemon daemons like this:
However, on some occasions the behavior is much different:
In the second example only "node_b" and "node_c" appear to be echoing these events, but what is additionally concerning is the retry values do not sync up correctly. ie. at 21:41:37 node_b thought this was the 55th retry while node_c thought it was the 63rd.
Here is a
mon node statusI looked through the troubleshooting notes and did get identical hashes for this command:
mon node ctrl --type=peer -- mon oconf hashBut my object.cache files do not have the same hash value. I looked through the files side-by-side (and with diff) and it looks like Naemon is listing the 'members' of certain host/service/etc groups in a randomized order, which throws the hashes off from each other.I looked at the merlin database and found that the 'report_data' table had wildly different rows of data. Could have been the result of testing the system. So I truncated that table and started over ... now all three have roughly the same number of rows ... 612, 619, 618. But that hasn't really resolved the problem with what's getting logged by Naemon.
I did a specific query for the device in question "pdu-f10" in the report_data table, and all three databases have the same exact info ... 'timestamp' and 'retry' are all consistent ... but the 'id' for each row is different (which I guess makes sense since it's auto-increment). So one Naemon node logs the right retry value, another one logs a retry value that's less, and the other one logs nothing at all.
Again my sincere apologies for posting this as an 'issue' and not a general support request elsewhere!