bug: Jail reason consensus not achieved

[cosmoscats]

What is happening?

Section description

Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.

Our node goes to Jail every day few times a day. According to the latest data, it almost always goes to Jail because of ETH RPC. I have the opportunity to install 10 different RPC providers for testing.

Paloma and pigeon versions and logs

Section description

Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.

palomad version v2.3.2

pigeon version App version: v2.3.1 Build commit hash: 5f6f4bcaa645e0f9d8530b24cd1a4cd74b79e857

i will attach latest latest jail log paloma-jail-24-10-2024.txt

How to reproduce?

Section description

Please write detailed steps of what you were doing for this bug to appear.

Unjail, wait for sometime and in 24 hours or less node will be jailed

What is the expected behaviour?

Section description

If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.

-

[taariq]

Logs added

Oct 24 20:27:28 cc1 palomad[2745431]: 8:27PM INF put message into consensus queue message-id=350030 module=x/palomaconsensus msg="hexAddresses:\"0x1177806cD88b0BC0eA363bEd3C64f8314361b162\" hexAddresses:\"0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb\" hexAddresses:\"0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A\" hexAddresses:\"0x35c1cC4E3CD4624Ac177c167D2427e9C95336535\" hexAddresses:\"0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2\" hexAddresses:\"0x1799D68d773f9E03Baca79cd1d836Ba026B4742A\" hexAddresses:\"0x43E218f96A567DC26C6e65fCabF1fDE26Af69444\" hexAddresses:\"0x443266026738061972012e62D5Ecd9D98da8B6F4\" hexAddresses:\"0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc\" hexAddresses:\"0xa228292447064D5818BbC80b577C91f5212F9355\" hexAddresses:\"0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a\" hexAddresses:\"0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2\" hexAddresses:\"0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42\" hexAddresses:\"0x2d91F6C502289eA4d7F9253C45788109377b95bD\" hexAddresses:\"0x732fBb6018F2cB5b844055C9EB567447a3132833\" hexAddresses:\"0x19f5911E4cA69E30449aD6BB71De341F01F118BB\" hexAddresses:\"0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca\" hexAddresses:\"0x63f55bc560E981d53E1f5bb3643e3a96D26fc635\" hexAddresses:\"0x6dc59EE4bdFa2C791229004f29b08F783491a934\" hexAddresses:\"0x14Aa448C2C918C4427c5671028b63BC17f6132d5\" hexAddresses:\"0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535\" hexAddresses:\"0x7Bd1A3270570b65F264895C2F86E631fDc497545\" valAddresses:\"\001\276s\021\343\227\312Q\246G*\224\363(\340K\273\350\006\" valAddresses:\"\002}0\274a\2108\266M\347f4\001<\236Jp\365bx\" valAddresses:\"\t\007\3044\210\241\355[\370\347\010W15\206\207(-h.\" valAddresses:\"\t\231\212\337\024\375x\026BE\3600>9\322\357\333\362\316\345\" valAddresses:\"\021R\201\333\020\021\310\304-\261\234\366\203F\\313\\363\\344\\322\" valAddresses:\"V5{\\265e\\371\\017\\023\\205\\332J\\335\\205\\245\\275\\342\\306q\\205G\" valAddresses:\"[\\\\\\250\\24435\\325\\313E\\262Z\\363\\177g\\244i64\\242}\" valAddresses:\"\\\\8y\\256\\374\\200sQ~\\331\\222\\345\\026+\\303X\\016\\001x\\032\" valAddresses:\"]:W\\321\\311\\303)o\\274S\\214s)\\260\\255\\266%<\\214\\004\" valAddresses:\"e%\\315\\342\\033\\245\\217\\234\\232H\\022S\\345O\\321\\331\\2446\\340\\332\" valAddresses:\"v\\352C\\222\\341\\t\\270\\335J\\216c\\207\\344\\220\\223\\275\\336U\\355\\224\" valAddresses:\"y\\024\\257H\\255yC\\213d\\304\\212\\230\\244b\\331\\034\\213\\327\\333\\031\" valAddresses:\"y~y\\006\\2421P~7+\\235\\350\\007\\\"\\002\\311\\325\\274\\344\\024\" valAddresses:\"y\\360\\311q\\3445\\352\\333\\313\\314$\\3268\\220\\324(\\375t\\326~\" valAddresses:\"\\200\\242)\200>\033\034\252\305\261\264\332\326\007t\372\304\300\212\" valAddresses:\"\203\273Ka%\307\350\323(\2013\361A\2274\003S1\004\256\" valAddresses:\"\203\377\331\332\361\215\202<\215R\247\331\332\203d9r\332\035\" valAddresses:\"\217!\373\274\212RS9\264\203\211\242\367\001\247\356\203\220]T\" valAddresses:\"\242\030KV&\253\364\317[\r\250\322\232\001<X7\211\206Q\" valAddresses:\"\27471u\333\237xB\200\3260\244\r\253aj\236\035\247\251\" valAddresses:\"\301\370kXOx&:\001/6rG\247w\3666G\024\" valAddresses:\"\3459+\360`_Mf-=\002\020\261\321\334\240!\263@P\" fromBlockTime: " queue-type-name=evm/eth-main/validators-balances Oct 24 20:27:31 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm Oct 24 20:27:31 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=147956415466737 Oct 24 20:27:33 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=147956415466737 Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf Oct 24 20:27:35 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=379703560579887 Oct 24 20:27:36 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=379703560579887 Oct 24 20:27:38 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x Oct 24 20:27:38 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=560504217079887 Oct 24 20:27:39 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=560504217079887 Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1qf7np0rp3qutvn08vc6qz0y7ffc02cncwdvdtd Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743 Oct 24 20:27:41 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=700112818385852 Oct 24 20:27:42 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g Oct 24 20:27:42 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149 Oct 24 20:27:44 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149 Oct 24 20:27:45 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=737193758269149 Oct 24 20:27:46 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze Oct 24 20:27:46 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=865190075873055 Oct 24 20:27:48 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=865190075873055 Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1sz3zjcyq8cd3e2k9kx6d44s8wnavfsy2837nje Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper108cvju0yxh4dhj7vyntr3yx59r7hf4n7kyzhu4 Oct 24 20:27:50 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1052826268020914 Oct 24 20:27:51 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1052826268020914 Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1pxvc4hc5l4upvsj97qcruwwjaldl9nh9v2k3z8 Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1u5ujhurqtaxkvtfaqggtr5wu5qsmxszs9hrvxe Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs Oct 24 20:27:53 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"792824356058927"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1430325117777092 Oct 24 20:27:54 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv Oct 24 20:27:54 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"912838586458895"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1550339348177060 Oct 24 20:27:56 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1t5a905wfcv5kl0zn33ejnv9dkcjnerqy9mwutc Oct 24 20:27:56 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1555622151779682 Oct 24 20:27:58 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1555622151779682 Oct 24 20:27:59 cc1 palomad[2745431]: 8:27PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2 Oct 24 20:27:59 cc1 palomad[2745431]: 8:27PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"985629872353235"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1623130634071400 Oct 24 20:28:01 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"985629872353235"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1623130634071400 Oct 24 20:28:02 cc1 palomad[2745431]: 8:28PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy Oct 24 20:28:02 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1053007719853992"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1690508481572157 Oct 24 20:28:21 cc1 palomad[2745431]: 8:28PM INF added message evidence. chain-reference-id=eth-main chain-type=evm message-id=350030 module=x/palomaconsensus queue-type-name=evm/eth-main/validators-balances validator=palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta Oct 24 20:28:21 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482 Oct 24 20:28:23 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482 Oct 24 20:28:24 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482 Oct 24 20:28:26 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482 Oct 24 20:28:27 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482 Oct 24 20:28:29 cc1 palomad[2745431]: 8:28PM ERR Consensus not achieved. error="evm: consensus not achieved" component=attest-message distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"1172876672276317"} module=x/evm msg-id=350030 msg-nonce="\x00\x00\x00\x00\x00\x05WN" total-shares=1995186409183528 total-votes=1810377433994482 Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF Removed message from queue block_height=25891550 module=server msg={"bytesToSign":"o1Pnaq+DEtinxt97XiWwf5B4W7QMLOUZj2xEQeq3Xxg=","errorData":null,"gasEstimate":"0","id":"350030","msg":{"@type":"/palomachain.paloma.evm.ValidatorBalancesAttestation","assignee":"","fromBlockTime":"2024-10-24T20:27:27.091967665Z","hexAddresses":["0x1177806cD88b0BC0eA363bEd3C64f8314361b162","0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb","0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A","0x35c1cC4E3CD4624Ac177c167D2427e9C95336535","0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2","0x1799D68d773f9E03Baca79cd1d836Ba026B4742A","0x43E218f96A567DC26C6e65fCabF1fDE26Af69444","0x443266026738061972012e62D5Ecd9D98da8B6F4","0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc","0xa228292447064D5818BbC80b577C91f5212F9355","0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a","0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2","0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42","0x2d91F6C502289eA4d7F9253C45788109377b95bD","0x732fBb6018F2cB5b844055C9EB567447a3132833","0x19f5911E4cA69E30449aD6BB71De341F01F118BB","0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca","0x63f55bc560E981d53E1f5bb3643e3a96D26fc635","0x6dc59EE4bdFa2C791229004f29b08F783491a934","0x14Aa448C2C918C4427c5671028b63BC17f6132d5","0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535","0x7Bd1A3270570b65F264895C2F86E631fDc497545"],"valAddresses":["palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","palomavaloper1qf7np0rp3qutvn08vc6qz0y7ffc02cncwdvdtd","palomavaloper1pyrugdyg58k4h788pptnzdvxsu5z66pwhu69gs","palomavaloper1pxvc4hc5l4upvsj97qcruwwjaldl9nh9v2k3z8","palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj","palomavaloper12c6hhdt9ly838pw6ftwctfdautr8rp288atqnf","palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz","palomavaloper1tsu8nthuspe4zlkejtj3v27rtq8qz7q6983zt2","palomavaloper1t5a905wfcv5kl0zn33ejnv9dkcjnerqy9mwutc","palomavaloper1v5jumcsm5k8eexjgzff72n73mxjrdcx6xu2e9x","palomavaloper1wm4y8yhppxud6j5wvwr7fyynhh09tmv5fy845g","palomavaloper10y227j9d09pckexy32v2gckerj9a0kcewgf7xy","palomavaloper109l8jp4zx9g8udetnh5qwgsze82meeq5yq06ta","palomavaloper108cvju0yxh4dhj7vyntr3yx59r7hf4n7kyzhu4","palomavaloper1sz3zjcyq8cd3e2k9kx6d44s8wnavfsy2837nje","palomavaloper1swa5kcf9cl5dx2ypx0c5r9e5qdfnzp9w0yj7uv","palomavaloper1s0lankh33kprer2l22nank5rvsuh9ksafx3mze","palomavaloper13uslh0y22ffnndyr3x30wqd8a6peqh25m8p743","palomavaloper15gvyk43x406v7kcd4rff5qfutqmcnpj3w9wpnm","palomavaloper1hsmnzawmnauy9qxkxzjqm2mpd20pmfaf5uj30x","palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q","palomavaloper1u5ujhurqtaxkvtfaqggtr5wu5qsmxszs9hrvxe"]},"nonce":"AAAAAAAFV04=","publicAccessData":"AQ==","signData":[],"valsetID":"0"} msg-id=350030

[cosmoscats]

go version go1.21.0 linux/amd64 maybe this can be the issue? do i need to update go version on server?

[maharifu]

The jail reason is here: Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg

One of the issues here is some pigeons are having RPC issues and returning incomplete balances information, which makes consensus harder to achieve. We can see that here distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"}. This means we have two different values for the same request, which in this request means some pigeons failed to get balances for all validators.

However, on the logs, we can see

Oct 24 20:26:50 mainnet-validator palomad[1574145]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"eth-main","msg.args.error":"failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","time":"2024-10-24T20:26:50Z","message":"error attesting messages"}

So this is an issue of failing to broadcast tx to paloma, a duplicate of https://github.com/VolumeFi/paloma/issues/2259

@cosmoscats I don't think the go version is the issue but you should still update it, especially if you are compiling your own binary. Are you using pigeon operator keys (like detailed here)?

[cosmoscats]

@maharifu - i will try to update go version.

When we starting to use pigeon operator keys - we go straight to the jail much faster, like in 1 hour or so (situation is same like Nodes Guru described in discord - "out of gas" We have disabled it for now.

[cosmoscats]

still no info about? https://github.com/VolumeFi/paloma/issues/2259