Pigeon issue - Failed to broadcast TX

[maharifu]

Pigeon is failing to send a tx to paloma, causing a validator to be jailed.

Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=error msg="Failed to execute process: failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get incl>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="updating external chain infos" component=update-external-chain-infos x-correlation-id=cs3rktepv0isuv8inv40
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="signing transaction" component=send-msgs key=cc2 x-correlation-id=cs3rjvepv0isuv8inlmg
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0x97ce46eB1137D839AD3027C68583C64d8B0c185C chain-reference-id=arbitrum-main chain->
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0x937506c931B2aB4c3aB2D114a2Cc5716f183A0e6 chain-reference-id=base-main chain-trai>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0xAcA6cDCFCBbccA070c5F18535F1c5cbaE2B4142E chain-reference-id=bnb-main chain-trait>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0x1177806cD88b0BC0eA363bEd3C64f8314361b162 chain-reference-id=eth-main chain-trait>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0xB66D86b4f926F97D946466027d950552890b3cb0 chain-reference-id=gnosis-main chain-tr>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0xB924F36582eB10e1388Fe7331E84dD42DdE0309d chain-reference-id=matic-main chain-tra>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=info msg="adding chain info to payload" acc-address=0xD96C286d07309dE363C1BDB54815234FF5a3bE85 chain-reference-id=op-main chain-traits>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=warning msg="Failed to broadcast TX." error="out of gas" sync-result="&{11  out of gas in location: ValuePerByte; gasWanted: 0, gasUse>
Oct 10 11:28:53 cc1 pigeon[2701589]: time="2024-10-10T11:28:53Z" level=error msg="error returned in process loop" error="failed to broadcast tx: out of gas" x-correlation-id=cs3rjvepv0isuv8inlmg

On our validator logs, we can find only two validators reporting this issue:

Oct 10 12:46:11 mainnet-validator palomad[1013801]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"eth-main","msg.args.error":"failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1c8uxkkz00qn97wsp9um8y3a8wlmrv3c546jw4q","time":"2024-10-10T12:46:11Z","message":"error attesting messages"}
Oct 10 13:01:06 mainnet-validator palomad[1013801]: {"level":"error","module":"server","module":"x/paloma","msg.args.chain-reference-id":"eth-main","msg.args.error":"failed to broadcast tx: timed out after: 60000000000; timed out after waiting for tx to get included in the block","component":"pigeon-status-update","status":"error attesting messages","sender":"palomavaloper1qxl8xy0rjl99zhaxgu4ffuegup9mh6qxdmcwkg","time":"2024-10-10T13:01:06Z","message":"error attesting messages"}

HW info from Cosmos Cats:

We are using own hardware server with ubuntu 22.04 installed
i9-14900K Intel Core
Z790 Gigabyte Aorus Master X
Corsair 64GB DDR5 Vengeance (2x32GB) 6600 CL32
4TB Kingston KC3000  & Samsung 990 Pro

[taariq]

@maharifu will you put a tracking ticket on Paloma? We'll invite other validators to add comments there.