Recently we had a live fuckup where one of the MongoDB hosts (mongodb-1) changed its IP address and got kicked out of the replica set because it was hardwired there with its old IP address.
MongoDB-1 became PRIMARY on its own, while mongodb-2 and -3 elected a new PRIMARY and continued operation normally.
rs.state() on mongodb-1 showed no problems.
rs.state() on mongodb-2 showed mongodb-1 as unreachable / not healthy.
However, the icinga check "replset_state" was green on all three hosts although there was a big fuckup.
So I think replset_state check should consider the state of all hosts in the cluster, not just the "mystate".
Recently we had a live fuckup where one of the MongoDB hosts (mongodb-1) changed its IP address and got kicked out of the replica set because it was hardwired there with its old IP address. MongoDB-1 became PRIMARY on its own, while mongodb-2 and -3 elected a new PRIMARY and continued operation normally.
rs.state() on mongodb-1 showed no problems. rs.state() on mongodb-2 showed mongodb-1 as unreachable / not healthy.
However, the icinga check "replset_state" was green on all three hosts although there was a big fuckup. So I think replset_state check should consider the state of all hosts in the cluster, not just the "mystate".