Closed D-Elektra closed 1 year ago
aditya-manit
commented
1 year ago I think number of calls can be optimized, my dashboard tells me that in last 30 days 1.6M calls were made by pigeon, this is a lot in comparison to some other cross chain projects, usually its more in the range of 10k to 40k requests per month. Maybe we can introduced some sort of batching
taariq
commented
1 year ago Thanks @aditya-manit. You are correct. We really want to optimize the number of calls. No need for so many.
byte-bandit
commented
1 year ago This might have been related to the failing compass EVM deployment to eth-main. @D-Elektra Does this issue still persist or have calls declined again since 9/14 ish?
D-Elektra
commented
1 year ago Sep 19 05:24:57 Ubuntu-2004-focal-64-minimal-hwe pigeon[3724837]: time="2023-09-19T05:24:57+02:00" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: 429 Too Many Requests: {\"code\":-32007,\"message\":\"25/second request limit reached - reduce calls per second or upgrade your account at quicknode.com\"}\n" message-ids="[71676]" messages-to-attest="[71676]" queue-name=evm/bnb-main/validators-balances x-correlation-id=ck4h92db1gjhbael00l0
One week = 8m requests from Testnet (BNB)
Finished.
I use paid one BNB for Mainnet. Its 14m already consumed.
taariq
commented
1 year ago Thanks @D-Elektra. This is very helpful.
taariq
commented
1 year ago @byte-bandit I am guessing we may want to look at attestation functionality and volume.
D-Elektra
commented
1 year ago 7 days of testnet BNB requests = 9.8m
byte-bandit
commented
1 year ago @D-Elektra
Thanks for submitting the data. We believe this is caused by the changes made to Pigeon during the v1.8.0 upgrade. We have a merged PR that will relax the amount of calls made including more information on the background here: https://github.com/palomachain/pigeon/pull/282
It should be fixed with the next iteration of Pigeon.
What is happening?
Endpoints cannot handle the amount of requests and pigeon getting jailed.
Since the last upgrade for 1.8.0 amount of requests for all endpoints grew a lot. x3 more than before. I have a subscription in Quicknodes so I kept it only for Testnet and Messenger chains. But within 3 days I consumed more than 11m requests.
So I decided to move most of the endpoints to Alchemy and other accounts of Quicknodes to avoid limitation in a few days. Optimism in Alchemy worked well for a day, but I got too many requests per second even on the paid account of Alchemy.
So again I had to change it back to Quicknodes for monitoring if it can work well. BUT!!! After few hours I got limitation in Quicknodes as well. It was a zero usage and in a few hours it was over the limit "We hope you're enjoying your experience with QuickNode. We noticed that your account has reached its endpoint limit of 1 endpoint." This is the Optimism one.
Paloma and pigeon versions and logs
versions and logs
palomad version 1.8.0. pigeon version App version: v1.8.1 Build commit hash: 2d246d7716522edcdff46912402c8624ee777f4c. loop-timeout: 5s health-check-port: 5757 paloma: chain-id: messenger call-timeout: 20s keyring-dir: ~/.paloma keyring-pass-env-name: PALOMA_KEYRING_PASS keyring-type: os signing-key: DiElektra base-rpc-url: http://localhost:26657 gas-adjustment: 1.5 gas-prices: 0.001ugrain account-prefix: paloma evm: eth-main: chain-id: 1 base-rpc-url: ${ETH_RPC_URL} keyring-pass-env-name: ETH_PASSWORD signing-key: ${ETH_SIGNING_KEY} keyring-dir: ~/.pigeon/keys/evm/eth-main gas-adjustment: 2 tx-type: 2 bnb-main: chain-id: 56 base-rpc-url: ${BNB_RPC_URL} keyring-pass-env-name: BNB_PASSWORD signing-key: ${BNB_SIGNING_KEY} keyring-dir: ~/.pigeon/keys/evm/bnb-main gas-adjustment: 1 tx-type: 0 matic-main: chain-id: 137 base-rpc-url: ${MATIC_RPC_URL} keyring-pass-env-name: MATIC_PASSWORD signing-key: ${MATIC_SIGNING_KEY} keyring-dir: ~/.pigeon/keys/evm/matic-main gas-adjustment: 2 kava-main: chain-id: 2222 base-rpc-url: ${KAVA_RPC_URL} keyring-pass-env-name: KAVA_PASSWORD signing-key: ${KAVA_SIGNING_KEY} keyring-dir: /root/.pigeon/keys/evm/kava-main gas-adjustment: 2 tx-type: 2 base-main: chain-id: 8453 base-rpc-url: ${BASE_RPC_URL} keyring-pass-env-name: BASE_PASSWORD signing-key: ${BASE_SIGNING_KEY} keyring-dir: ~/.pigeon/keys/evm/base-main gas-adjustment: 1.2 tx-type: 2 arb-main: chain-id: 42161 base-rpc-url: ${ARB_RPC_URL} keyring-pass-env-name: ARB_PASSWORD signing-key: ${ARB_SIGNING_KEY} keyring-dir: /root/.pigeon/keys/evm/arb-main gas-adjustment: 2 tx-type: 2Logs of errors
Pigeon logs
Sep 14 19:59:59 ns31451738 pigeon[3622585]: time="2023-09-14T19:59:59Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: not found\n" message-ids="[98797]" messages-to-attest="[98797]" queue-name=evm/bnb-main/evm-turnstone-message x-correlation-id=ck1mcfu4ced8adc949fg Sep 14 19:59:59 ns31451738 pigeon[3622585]: time="2023-09-14T19:59:59Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: not found\n" Sep 14 20:00:00 ns31451738 pigeon[3622585]: time="2023-09-14T20:00:00Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: not found\n" message-ids="[98797]" messages-to-attest="[98797]" queue-name=evm/bnb-main/evm-turnstone-message x-correlation-id=ck1mcg64ced8adc949hg Sep 14 20:00:00 ns31451738 pigeon[3622585]: time="2023-09-14T20:00:00Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: not found\n" Sep 14 20:00:00 ns31451738 pigeon[3622585]: time="2023-09-14T20:00:00Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: not found\n" message-ids="[98797]" messages-to-attest="[98797]" queue-name=evm/bnb-main/evm-turnstone-message x-correlation-id=ck1mcg64ced8adc949jg Sep 14 20:00:00 ns31451738 pigeon[3622585]: time="2023-09-14T20:00:00Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: not found\n" Sep 14 20:26:26 ns31451738 pigeon[3622585]: time="2023-09-14T20:26:26Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: Post \"https://autumn-ancient-moon.optimism.discover.quiknode.pro/7169274ce0298b54a8674918a420fe4f4bbd51a7/\": read tcp 141.94.97.169:38188->144.24.46.35:443: read: connection reset by peer\n" message-ids="[98840]" messages-to-attest="[98840]" queue-name=evm/op-main/validators-balances x-correlation-id=ck1moku4ced8adc97asg Sep 14 20:26:26 ns31451738 pigeon[3622585]: time="2023-09-14T20:26:26Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: Post \"https://autumn-ancient-moon.optimism.discover.quiknode.pro/7169274ce0298b54a8674918a420fe4f4bbd51a7/\": read tcp 141.94.97.169:38188->144.24.46.35:443: read: connection reset by peer\n" Sep 14 20:26:28 ns31451738 pigeon[3622585]: time="2023-09-14T20:26:28Z" level=error msg="couldn't get events" action=handle-gravity-batch-send-events chain-reference-id=op-main error="Post \"https://autumn-ancient-moon.optimism.discover.quiknode.pro/7169274ce0298b54a8674918a420fe4f4bbd51a7/\": dial tcp 144.24.46.35:443: connect: connection refused" x-correlation-id=ck1mol64ced8adc97av0 Sep 14 20:26:28 ns31451738 pigeon[3622585]: time="2023-09-14T20:26:28Z" level=error msg="error returned in process loop" err="Post \"https://autumn-ancient-moon.optimism.discover.quiknode.pro/7169274ce0298b54a8674918a420fe4f4bbd51a7/\": dial tcp 144.24.46.35:443: connect: connection refused" Sep 14 20:41:41 ns31451738 pigeon[3622585]: time="2023-09-14T20:41:41Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98846 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '266890' : unknown request\n" message-ids="[98846]" messages-to-attest="[98846]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1mvse4ced8adc99a90 Sep 14 20:41:41 ns31451738 pigeon[3622585]: time="2023-09-14T20:41:41Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98846 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '266890' : unknown request\n" Sep 14 20:42:05 ns31451738 pigeon[3622585]: time="2023-09-14T20:42:05Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98848 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267107' : unknown request\n" message-ids="[98848]" messages-to-attest="[98848]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1n01e4ced8adc99agg Sep 14 20:42:05 ns31451738 pigeon[3622585]: time="2023-09-14T20:42:05Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98848 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267107' : unknown request\n" Sep 14 20:42:24 ns31451738 pigeon[3622585]: time="2023-09-14T20:42:24Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98850 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '266904' : unknown request\n" message-ids="[98850]" messages-to-attest="[98850]" queue-name=evm/op-main/validators-balances x-correlation-id=ck1n07e4ced8adc99an0 Sep 14 20:42:24 ns31451738 pigeon[3622585]: time="2023-09-14T20:42:24Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98850 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '266904' : unknown request\n" Sep 14 20:49:17 ns31451738 pigeon[3622585]: time="2023-09-14T20:49:17Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" message-ids="[98851]" messages-to-attest="[98851]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1n3gm4ced8adc9a8o0 Sep 14 20:49:17 ns31451738 pigeon[3622585]: time="2023-09-14T20:49:17Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" Sep 14 21:13:07 ns31451738 pigeon[3622585]: time="2023-09-14T21:13:07Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98866 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" message-ids="[98866]" messages-to-attest="[98866]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1nelm4ced8adc9d75g Sep 14 21:13:07 ns31451738 pigeon[3622585]: time="2023-09-14T21:13:07Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98866 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" Sep 14 21:14:16 ns31451738 pigeon[3622585]: time="2023-09-14T21:14:16Z" level=error msg="couldn't get events" action=handle-gravity-batch-send-events chain-reference-id=matic-main error="Post \"https://polygon-mainnet.g.alchemy.com/v2/aI-HUfyi87TIb-9Brk88YJHRzrGWnUOc\": read tcp [2001:41d0:403:48a9::]:36980->[2606:4700::6810:c0]:443: read: connection reset by peer" x-correlation-id=ck1nfa64ced8adc9damg Sep 14 21:14:16 ns31451738 pigeon[3622585]: time="2023-09-14T21:14:16Z" level=error msg="error returned in process loop" err="Post \"https://polygon-mainnet.g.alchemy.com/v2/aI-HUfyi87TIb-9Brk88YJHRzrGWnUOc\": read tcp [2001:41d0:403:48a9::]:36980->[2606:4700::6810:c0]:443: read: connection reset by peer" Sep 14 21:36:30 ns31451738 pigeon[3622585]: time="2023-09-14T21:36:30Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: couldn't find block: block has not yet been generated\n" message-ids="[98883]" messages-to-attest="[98883]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1npiu4ced8adc9g5sg Sep 14 21:36:30 ns31451738 pigeon[3622585]: time="2023-09-14T21:36:30Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: couldn't find block: block has not yet been generated\n" Sep 14 21:36:54 ns31451738 pigeon[3622585]: time="2023-09-14T21:36:54Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98883 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267487' : unknown request\n" message-ids="[98883]" messages-to-attest="[98883]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1npnm4ced8adc9g660 Sep 14 21:36:54 ns31451738 pigeon[3622585]: time="2023-09-14T21:36:54Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98883 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267487' : unknown request\n" Sep 14 21:37:13 ns31451738 pigeon[3622585]: time="2023-09-14T21:37:13Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98885 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267274' : unknown request\n" message-ids="[98885]" messages-to-attest="[98885]" queue-name=evm/op-main/validators-balances x-correlation-id=ck1nptm4ced8adc9g6d0 Sep 14 21:37:13 ns31451738 pigeon[3622585]: time="2023-09-14T21:37:13Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98885 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267274' : unknown request\n" Sep 14 22:46:57 ns31451738 pigeon[3622585]: time="2023-09-14T22:46:57Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" message-ids="[98926]" messages-to-attest="[98926]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1oqlm4ced8adc9p7n0 Sep 14 22:46:57 ns31451738 pigeon[3622585]: time="2023-09-14T22:46:57Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" Sep 14 23:44:49 ns31451738 pigeon[3622585]: time="2023-09-14T23:44:49Z" level=error msg="couldn't get events" action=handle-gravity-batch-send-events chain-reference-id=bnb-main error="Post \"https://floral-ancient-arrow.bsc.quiknode.pro/cd0ceaf0a3ad559adee04c9766a51e3541cad243/\": read tcp 141.94.97.169:51348->130.61.117.76:443: read: connection reset by peer" x-correlation-id=ck1plrm4ced8adca0ga0 Sep 14 23:44:49 ns31451738 pigeon[3622585]: time="2023-09-14T23:44:49Z" level=error msg="error returned in process loop" err="Post \"https://floral-ancient-arrow.bsc.quiknode.pro/cd0ceaf0a3ad559adee04c9766a51e3541cad243/\": read tcp 141.94.97.169:51348->130.61.117.76:443: read: connection reset by peer" Sep 15 00:13:03 ns31451738 pigeon[3622585]: time="2023-09-15T00:13:03Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98981 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" message-ids="[98981]" messages-to-attest="[98981]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1q2vm4ced8adca43fg Sep 15 00:13:03 ns31451738 pigeon[3622585]: time="2023-09-15T00:13:03Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98981 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" Sep 15 00:13:24 ns31451738 pigeon[3622585]: time="2023-09-15T00:13:24Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" message-ids="[98983]" messages-to-attest="[98983]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1q33u4ced8adca43ng Sep 15 00:13:24 ns31451738 pigeon[3622585]: time="2023-09-15T00:13:24Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" Sep 15 00:13:45 ns31451738 pigeon[3622585]: time="2023-09-15T00:13:45Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98985 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267094' : unknown request\n" message-ids="[98985]" messages-to-attest="[98985]" queue-name=evm/op-main/validators-balances x-correlation-id=ck1q3964ced8adca43ug Sep 15 00:13:45 ns31451738 pigeon[3622585]: time="2023-09-15T00:13:45Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98985 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267094' : unknown request\n" Sep 15 00:20:56 ns31451738 pigeon[3622585]: time="2023-09-15T00:20:56Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: couldn't find block: block has not yet been generated\n" message-ids="[98988]" messages-to-attest="[98988]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1q6m64ced8adca51qg Sep 15 00:20:56 ns31451738 pigeon[3622585]: time="2023-09-15T00:20:56Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: couldn't find block: block has not yet been generated\n" Sep 15 00:21:01 ns31451738 pigeon[3622585]: time="2023-09-15T00:21:01Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: couldn't find block: block has not yet been generated\n" message-ids="[98988]" messages-to-attest="[98988]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1q6q64ced8adca522g Sep 15 00:21:01 ns31451738 pigeon[3622585]: time="2023-09-15T00:21:01Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: couldn't find block: block has not yet been generated\n" Sep 15 00:21:33 ns31451738 pigeon[3622585]: time="2023-09-15T00:21:33Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" message-ids="[98990]" messages-to-attest="[98990]" queue-name=evm/op-main/validators-balances x-correlation-id=ck1q6re4ced8adca5240 Sep 15 00:21:33 ns31451738 pigeon[3622585]: time="2023-09-15T00:21:33Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" Sep 15 00:36:31 ns31451738 pigeon[3622585]: time="2023-09-15T00:36:31Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98996 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" message-ids="[98996]" messages-to-attest="[98996]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1qe0e4ced8adca6u4g Sep 15 00:36:31 ns31451738 pigeon[3622585]: time="2023-09-15T00:36:31Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98996 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" Sep 15 00:36:56 ns31451738 pigeon[3622585]: time="2023-09-15T00:36:56Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98998 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267297' : unknown request\n" message-ids="[98998]" messages-to-attest="[98998]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1qe3u4ced8adca6ubg Sep 15 00:36:56 ns31451738 pigeon[3622585]: time="2023-09-15T00:36:56Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 98998 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267297' : unknown request\n" Sep 15 00:49:19 ns31451738 pigeon[3622585]: time="2023-09-15T00:49:19Z" level=error msg="couldn't get events" action=handle-gravity-send-to-paloma-events chain-reference-id=bnb-main error="Post \"https://floral-ancient-arrow.bsc.quiknode.pro/cd0ceaf0a3ad559adee04c9766a51e3541cad243/\": read tcp 141.94.97.169:42628->136.244.66.101:443: read: connection reset by peer" x-correlation-id=ck1qk364ced8adca8k20 Sep 15 00:49:19 ns31451738 pigeon[3622585]: time="2023-09-15T00:49:19Z" level=error msg="error returned in process loop" err="Post \"https://floral-ancient-arrow.bsc.quiknode.pro/cd0ceaf0a3ad559adee04c9766a51e3541cad243/\": read tcp 141.94.97.169:42628->136.244.66.101:443: read: connection reset by peer" Sep 15 00:49:19 ns31451738 pigeon[3622585]: time="2023-09-15T00:49:19Z" level=error msg="couldn't get events" action=handle-gravity-batch-send-events chain-reference-id=bnb-main error="Post \"https://floral-ancient-arrow.bsc.quiknode.pro/cd0ceaf0a3ad559adee04c9766a51e3541cad243/\": dial tcp 136.244.66.101:443: connect: connection refused" x-correlation-id=ck1qk364ced8adca8k00 Sep 15 00:49:19 ns31451738 pigeon[3622585]: time="2023-09-15T00:49:19Z" level=error msg="error returned in process loop" err="Post \"https://floral-ancient-arrow.bsc.quiknode.pro/cd0ceaf0a3ad559adee04c9766a51e3541cad243/\": dial tcp 136.244.66.101:443: connect: connection refused" Sep 15 00:52:06 ns31451738 pigeon[3622585]: time="2023-09-15T00:52:06Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 99006 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" message-ids="[99006]" messages-to-attest="[99006]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1ql9u4ced8adca8veg Sep 15 00:52:06 ns31451738 pigeon[3622585]: time="2023-09-15T00:52:06Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 99006 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267080' : unknown request\n" Sep 15 00:52:28 ns31451738 pigeon[3622585]: time="2023-09-15T00:52:28Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 99008 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267297' : unknown request\n" message-ids="[99008]" messages-to-attest="[99008]" queue-name=evm/eth-main/validators-balances x-correlation-id=ck1qldm4ced8adca8vl0 Sep 15 00:52:28 ns31451738 pigeon[3622585]: time="2023-09-15T00:52:28Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 99008 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267297' : unknown request\n" Sep 15 00:52:46 ns31451738 pigeon[3622585]: time="2023-09-15T00:52:46Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 99010 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267094' : unknown request\n" message-ids="[99010]" messages-to-attest="[99010]" queue-name=evm/op-main/validators-balances x-correlation-id=ck1qlj64ced8adca8vt0 Sep 15 00:52:46 ns31451738 pigeon[3622585]: time="2023-09-15T00:52:46Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: rpc error: code = Unknown desc = rpc error: code = Unknown desc = failed to execute message; message index: 0: message id 99010 does not exist [cosmos/cosmos-sdk@v0.47.4/baseapp/baseapp.go:802] With gas wanted: '18446744073709551615' and gas used: '267094' : unknown request\n" Sep 15 00:59:46 ns31451738 pigeon[3622585]: time="2023-09-15T00:59:46Z" level=error msg="error attesting messages" action=attest error="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n" message-ids="[99011]" messages-to-attest="[99011]" queue-name=evm/base-main/validators-balances x-correlation-id=ck1qou64ced8adca9tv0 Sep 15 00:59:46 ns31451738 pigeon[3622585]: time="2023-09-15T00:59:46Z" level=error msg="error returned in process loop" err="there are 1 errors in the group:\nerror 1: transaction failed with code: 1\n"