Open cosmoscats opened 3 hours ago
Logs added
Oct 24 20:27:28 cc1 palomad[2745431]: 8:27PM INF put message into consensus queue message-id=350030 module=x/palomaconsensus msg="hexAddresses:\"0x1177806cD88b0BC0eA363bEd3C64f8314361b162\" hexAddresses:\"0x0DAaBB4FF60423Eb1F14cC6731394e098ad51bcb\" hexAddresses:\"0xC1009C72cC2519B0DD595C3E500C1293A862Aa6A\" hexAddresses:\"0x35c1cC4E3CD4624Ac177c167D2427e9C95336535\" hexAddresses:\"0x377D23948D41579f2c3cA40308e3bdd53f6dA7B2\" hexAddresses:\"0x1799D68d773f9E03Baca79cd1d836Ba026B4742A\" hexAddresses:\"0x43E218f96A567DC26C6e65fCabF1fDE26Af69444\" hexAddresses:\"0x443266026738061972012e62D5Ecd9D98da8B6F4\" hexAddresses:\"0x4a89f96fdff3C161937CFBC3e22d2E325612aaEc\" hexAddresses:\"0xa228292447064D5818BbC80b577C91f5212F9355\" hexAddresses:\"0xDEea5b069208e0eE37b630A3e7672FC50e8fE24a\" hexAddresses:\"0xAAc74d38c82c367b3dA7482E3Aecf6DD7A512eF2\" hexAddresses:\"0xeB784b37365C302C97C9Ec8cB0933cE344e6dE42\" hexAddresses:\"0x2d91F6C502289eA4d7F9253C45788109377b95bD\" hexAddresses:\"0x732fBb6018F2cB5b844055C9EB567447a3132833\" hexAddresses:\"0x19f5911E4cA69E30449aD6BB71De341F01F118BB\" hexAddresses:\"0x1ad90dB98da083E117D5F62a1673fC0f3A5930ca\" hexAddresses:\"0x63f55bc560E981d53E1f5bb3643e3a96D26fc635\" hexAddresses:\"0x6dc59EE4bdFa2C791229004f29b08F783491a934\" hexAddresses:\"0x14Aa448C2C918C4427c5671028b63BC17f6132d5\" hexAddresses:\"0xEe21301aF1d9562B5cBEdf520077Ea0a9bC9d535\" hexAddresses:\"0x7Bd1A3270570b65F264895C2F86E631fDc497545\" valAddresses:\"\001\276s\021\343\227\312Q\246G*\224\363(\340K\273\350\006\" valAddresses:\"\002}0\274a\2108\266M\347f4\001<\236Jp\365bx\" valAddresses:\"\t\007\3044\210\241\355[\370\347\010W15\206\207(-h.\" valAddresses:\"\t\231\212\337\024\375x\026BE\3600>9\322\357\333\362\316\345\" valAddresses:\"\021R\201\333\020\021\310\304-\261\234\366\203F\\313\\363\\344\\322\" valAddresses:\"V5{\\265e\\371\\017\\023\\205\\332J\\335\\205\\245\\275\\342\\306q\\205G\" valAddresses:\"[\\\\\\250\\24435\\325\\313E\\262Z\\363\\177g\\244i64\\242}\" valAddresses:\"\\\\8y\\256\\374\\200sQ~\\331\\222\\345\\026+\\303X\\016\\001x\\032\" valAddresses:\"]:W\\321\\311\\303)o\\274S\\214s)\\260\\255\\266%<\\214\\004\" valAddresses:\"e%\\315\\342\\033\\245\\217\\234\\232H\\022S\\345O\\321\\331\\2446\\340\\332\" valAddresses:\"v\\352C\\222\\341\\t\\270\\335J\\216c\\207\\344\\220\\223\\275\\336U\\355\\224\" valAddresses:\"y\\024\\257H\\255yC\\213d\\304\\212\\230\\244b\\331\\034\\213\\327\\333\\031\" valAddresses:\"y~y\\006\\2421P~7+\\235\\350\\007\\\"\\002\\311\\325\\274\\344\\024\" valAddresses:\"y\\360\\311q\\3445\\352\\333\\313\\314$\\3268\\220\\324(\\375t\\326~\" valAddresses:\"\\200\\242)
\200>\033\034\252\305\261\264\332\326\007t\372\304\300\212\" valAddresses:\"\203\273Ka%\307\350\323(\2013\361A\2274\003S1\004\256\" valAddresses:\"\203\377\331\332\361\215\202<\215R\247\331\332\203d9r\332\035\" valAddresses:\"\217!\373\274\212RS9\264\203\211\242\367\001\247\356\203\220]T\" valAddresses:\"\242\030KV&\253\364\317[\r\250\322\232\001<X7\211\206Q\" valAddresses:\"\27471u\333\237xB\200\3260\244\r\253aj\236\035\247\251\" valAddresses:\"\301\370kXOx&:\001/6rG\247w\3666G\024\" valAddresses:\"\3459+\360`_Mf-=\002\020\261\321\334\240!\263@P\" fromBlockTime:
go version go1.21.0 linux/amd64 maybe this can be the issue? do i need to update go version on server?
The jail reason is here:
Oct 24 20:36:51 cc1 palomad[2745431]: 8:36PM INF jailing a validator jail-time=2024-10-24T20:37:49Z module=x/valset reason="No evidence supplied for contentious message 350030" val-addr=palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg
One of the issues here is some pigeons are having RPC issues and returning incomplete balances information, which makes consensus harder to achieve. We can see that here distribution={"dc9f5115bc70d34c55aa7a1917f04a73764393c2a66e4d81401a96eb76c758ca":"637500761718165","e953b8678138b429195979e95c1d1badcb8f5ddcadf599fe962370f7f6e21e33":"918121390061517"}
. This means we have two different values for the same request, which in this request means some pigeons failed to get balances for all validators.
However, on the logs, we can see
Oct 24 20:26:50 mainnet-validator palomad[1574145]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"eth-main","msg.args.error":"failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","time":"2024-10-24T20:26:50Z","message":"error attesting messages"}
So this is an issue of failing to broadcast tx to paloma, a duplicate of https://github.com/VolumeFi/paloma/issues/2259
@cosmoscats I don't think the go version is the issue but you should still update it, especially if you are compiling your own binary. Are you using pigeon operator keys (like detailed here)?
@maharifu - i will try to update go version.
When we starting to use pigeon operator keys - we go straight to the jail much faster, like in 1 hour or so (situation is same like Nodes Guru described in discord - "out of gas" We have disabled it for now.
still no info about? https://github.com/VolumeFi/paloma/issues/2259
What is happening?
Section description
Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.Our node goes to Jail every day few times a day. According to the latest data, it almost always goes to Jail because of ETH RPC. I have the opportunity to install 10 different RPC providers for testing.
Paloma and pigeon versions and logs
Section description
Write down paloma version. Write down pigeon version. Copy and paste pigeon config file as well as relevant ENV variables.palomad version v2.3.2
pigeon version App version: v2.3.1 Build commit hash: 5f6f4bcaa645e0f9d8530b24cd1a4cd74b79e857
i will attach latest latest jail log paloma-jail-24-10-2024.txt
How to reproduce?
Section description
Please write detailed steps of what you were doing for this bug to appear.Unjail, wait for sometime and in 24 hours or less node will be jailed
What is the expected behaviour?
Section description
If you know, please write down what is the expected behaviour. If you don't know, that's ok. We can have a discussion in comments.-