Unexpected Jailing of Multiple Validators on Apr 13

[redwest88]

Summary of Bug

On April 13th, at 21:45:07, our validator (nibivalcons1elaj26jpnt0vzg2q53ggdmfzle9mskus3pufpq) was unexpectedly jailed along with two other validators. This incident is surprising because our records and monitoring tools indicated active participation up to block 5029202. Our validator could not have been jailed for downtime as per the Nibiru slashing window is 17500 / 35000 (50%), and yet the jailing occurred.

Nibid version

commit: 1ba22a79e36e7ed77e2e4503e72bfbbe5c609aa8
cosmos_sdk_version: v0.47.10
go: go version go1.21.5 linux/amd64
name: nibiru
server_name: nibid
version: 1.2.0

Below are the logs indicating our validator's participation up to the 5029202 block:

Following are the system logs capturing the jailing events for all three validators at the same time.:

Apr 13 21:45:07 RAM-NIBIRU nibid[110961]: 9:45PM INF validator jailed module=x/staking validator=nibivalcons12udeyt65x5qm972t0v28u5hy9j5ymfje3kzdcr
Apr 13 21:45:07 RAM-NIBIRU nibid[110961]: 9:45PM INF validator jailed module=x/staking validator=nibivalcons1xqvp5mhnz8e4fvwwmq0trd0370pep4uh0ru8m9
Apr 13 21:45:07 RAM-NIBIRU nibid[110961]: 9:45PM INF validator jailed module=x/staking validator=nibivalcons1elaj26jpnt0vzg2q53ggdmfzle9mskus3pufpq

Jail reason

We have also verified that the jailing was due to missing 5 blocks. However, according to the slashing policy and our participation logs, this jail does not seem to be justified. Our records show active participation until block 5029202. Only after block 5029203 our validator stop participating.

nibid query slashing signing-info $(nibid tendermint show-validator)
address: nibivalcons1elaj26jpnt0vzg2q53ggdmfzle9mskus3pufpq
index_offset: "1606259"
jailed_until: "1970-01-01T00:00:00Z"
missed_blocks_counter: "5"
start_height: "3422938"
tombstoned: false

A screenshot from our Grafana dashboard showing the uptime of our validator for the last 24 hours. The sharp drop coincides with a time when our validator was unexpectedly jailed, despite consistently high uptime before that.

Summary

We’d appreciate any info or tips from the Nibiru team on what caused this and how we can avoid it in the future, as there may be a potential discrepancy or issue in the network or jailing logic.

Best Regards, The F5 Nodes team

[k-yang]

Hi @redwest88 ,

Are you running an oracle?

[redwest88]

Hi @k-yang,

Yes, our pricefeeder account is nibi1d8xgp3pqkm7vvqtusj9myw7asddfarnd7knf2p