Relayer cannot finalize transactions on custom geth chain

shaspitz commented 9 months ago

Context

I'm running a POA geth sidechain on a single machine using this docker compose setup. We're testing hyperlane warp route bridging from Sepolia to and from our evm chain. See bridge setup here.

Problem

It seems a commit to main after Nov 20th 2023 has changed agent tx parameters such that transactions become stuck as pending on our geth chain. Using an older agent image from Nov 20th 2023, this behavior is not present and the relayer is able to submit txes to our chain.

To exemplify this issue I'll go through some steps..

Spin up relayer with recent image release

First I try spinning up our relayer using the most recent agent release: https://github.com/hyperlane-xyz/hyperlane-monorepo/releases/tag/agents-2023-12-14. The relayer initializes just fine, and I invoke a bridge transaction on Sepolia. The relayer picks up on this message, and attempts to deliver it to our sidechain. Logs show the relayer transaction stays pending. Note relayer account is 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6.

Logs:

2024-01-19 22:05:53   2024-01-20T06:05:53.267278Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118782..=5118783, num_logs: 2, estimated_time_to_sync: "synced"
2024-01-19 22:05:53     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:05:53     in hyperlane_base::contract_sync::ContractSync with label: "dispatched_messages", domain: "sepolia"
2024-01-19 22:05:53 
2024-01-19 22:05:58   2024-01-20T06:05:58.987294Z  INFO hyperlane_ethereum::tx: Dispatching transaction, to: 0x4426b0196dc89c67021a81872bcffaf6a4d8486c, data: 0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d05b4ea218ffedb1b9a8b85bd3e61483e2b82d5f4830fefc15898b0e44cada0d8000013e53157fc546c7ebe40a0220526899661918889629ecb7f2c2e829f825ef2a0c2732c82e44af95970faf93b403253ccaf08c443a7b1c630dcebfd2ecabdadfb219e1b5305092fb94f55ec9a3820ee3d7cc1bd044b5e7467c415fbcfbe4d76af38e779760eb282b70e2028028e58d93c734f9098a5d42c0f7cb24351918fc7c348e5511b00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000140100aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b900000000000000000000000000000000000000
2024-01-19 22:05:58     at chains/hyperlane-ethereum/src/tx.rs:38
2024-01-19 22:05:58     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x878c25e697df238a43c16478acfadfa4281b7d8b93dc15ea521dc6a5524b248c, version: 3, nonce: 5121, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b9 }, tx_gas_limit: Some(176104), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d05b4ea218ffedb1b9a8b85bd3e61483e2b82d5f4830fefc15898b0e44cada0d8000013e53157fc546c7ebe40a0220526899661918889629ecb7f2c2e829f825ef2a0c2732c82e44af95970faf93b403253ccaf08c443a7b1c630dcebfd2ecabdadfb219e1b5305092fb94f55ec9a3820ee3d7cc1bd044b5e7467c415fbcfbe4d76af38e779760eb282b70e2028028e58d93c734f9098a5d42c0f7cb24351918fc7c348e5511b
2024-01-19 22:05:58     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x878c25e697df238a43c16478acfadfa4281b7d8b93dc15ea521dc6a5524b248c, version: 3, nonce: 5121, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b9 } }
2024-01-19 22:05:58     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-19 22:05:58 
2024-01-19 22:05:59   2024-01-20T06:05:59.259536Z  INFO hyperlane_ethereum::tx: Dispatched tx, to: 0x4426b0196dc89c67021a81872bcffaf6a4d8486c, data: 0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d05b4ea218ffedb1b9a8b85bd3e61483e2b82d5f4830fefc15898b0e44cada0d8000013e53157fc546c7ebe40a0220526899661918889629ecb7f2c2e829f825ef2a0c2732c82e44af95970faf93b403253ccaf08c443a7b1c630dcebfd2ecabdadfb219e1b5305092fb94f55ec9a3820ee3d7cc1bd044b5e7467c415fbcfbe4d76af38e779760eb282b70e2028028e58d93c734f9098a5d42c0f7cb24351918fc7c348e5511b00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000140100aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b900000000000000000000000000000000000000, tx_hash: 0x9b6988dfdd60bbe107b17e6aa66ae491052a177b9dcd8fe7b51ecbf3f66bc3de
2024-01-19 22:05:59     at chains/hyperlane-ethereum/src/tx.rs:45
2024-01-19 22:05:59     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x878c25e697df238a43c16478acfadfa4281b7d8b93dc15ea521dc6a5524b248c, version: 3, nonce: 5121, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b9 }, tx_gas_limit: Some(176104), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d05b4ea218ffedb1b9a8b85bd3e61483e2b82d5f4830fefc15898b0e44cada0d8000013e53157fc546c7ebe40a0220526899661918889629ecb7f2c2e829f825ef2a0c2732c82e44af95970faf93b403253ccaf08c443a7b1c630dcebfd2ecabdadfb219e1b5305092fb94f55ec9a3820ee3d7cc1bd044b5e7467c415fbcfbe4d76af38e779760eb282b70e2028028e58d93c734f9098a5d42c0f7cb24351918fc7c348e5511b
2024-01-19 22:05:59     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x878c25e697df238a43c16478acfadfa4281b7d8b93dc15ea521dc6a5524b248c, version: 3, nonce: 5121, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b9 } }
2024-01-19 22:05:59     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-19 22:05:59 
2024-01-19 22:06:01   2024-01-20T06:06:01.557986Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118782..=5118784, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:06:01     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:06:01     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-19 22:06:01 
2024-01-19 22:06:06   2024-01-20T06:06:06.261707Z DEBUG ethers_providers::pending_transaction: Starting to poll pending tx 0x9b6988dfdd60bbe107b17e6aa66ae491052a177b9dcd8fe7b51ecbf3f66bc3de
2024-01-19 22:06:06     at /usr/local/cargo/git/checkouts/ethers-rs-3cecc26f7936b033/c9ced03/ethers-providers/src/pending_transaction.rs:175
2024-01-19 22:06:06     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x878c25e697df238a43c16478acfadfa4281b7d8b93dc15ea521dc6a5524b248c, version: 3, nonce: 5121, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b9 }, tx_gas_limit: Some(176104), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d05b4ea218ffedb1b9a8b85bd3e61483e2b82d5f4830fefc15898b0e44cada0d8000013e53157fc546c7ebe40a0220526899661918889629ecb7f2c2e829f825ef2a0c2732c82e44af95970faf93b403253ccaf08c443a7b1c630dcebfd2ecabdadfb219e1b5305092fb94f55ec9a3820ee3d7cc1bd044b5e7467c415fbcfbe4d76af38e779760eb282b70e2028028e58d93c734f9098a5d42c0f7cb24351918fc7c348e5511b
2024-01-19 22:06:06     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x878c25e697df238a43c16478acfadfa4281b7d8b93dc15ea521dc6a5524b248c, version: 3, nonce: 5121, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b7800000000000000000000000000000000000000000000000000000000000038b9 } }
2024-01-19 22:06:06     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-19 22:06:06 
2024-01-19 22:06:31   2024-01-20T06:06:31.252215Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118785..=5118786, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:06:31     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:06:31     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-19 22:06:31 
2024-01-19 22:07:01   2024-01-20T06:07:01.323292Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118787..=5118789, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:07:01     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:07:01     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-19 22:07:01 
2024-01-19 22:07:31   2024-01-20T06:07:31.403449Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118790..=5118791, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:07:31     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:07:31     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-19 22:07:31 
2024-01-19 22:08:01   2024-01-20T06:08:01.500991Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118792..=5118794, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:08:01     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:08:01     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"

As you can see the relayer submits a tx and prints Starting to poll pending tx 0x9b6988dfdd60bbe107b17e6aa66ae491052a177b9dcd8fe7b51ecbf3f66bc3de. Afterwards, the transaction never gets included in a block. To confirm the relayer's transaction is stuck as pending I made the following cast commands

cast nonce --rpc-url http://localhost:8545/ --block pending 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6
2
cast nonce --rpc-url http://localhost:8545/ 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6
1

These commands show the relayers tx with nonce=1 is stuck as pending.

Bring down relayer and cancel pending tx

Next I stopped the relayer container, and attempted to cancel the pending tx from the relayer's account

cast send --rpc-url http://localhost:8545/ --private-key <0xpk> --value 0 --gas-price <high value> --priority-gas-price ≤high value≥ --nonce 1 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6

This transaction gets confirmed

cast nonce --rpc-url http://localhost:8545 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6
2
cast nonce --rpc-url http://localhost:8545 --block pending 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6
2

Spin up relayer with working image from Nov 20th 2023

Next I spun up the relayer again, but this time using (this image (gcr.io/abacus-labs-dev/hyperlane-agent@sha256:854f92966eac6b49e5132e152cc58168ecdddc76c2d390e657b81bdaf1396af0) from this commit in main.

The relayer spins up successfully, processes the relevant pending message, and has it's transaction finalized on our sidechain. See logs below

2024-01-19 22:33:17 
2024-01-19 22:33:17   2024-01-20T06:33:17.550514Z DEBUG ethers_providers::pending_transaction: Checking receipt for pending tx 0xfb78a1dadd3818164737083ed88e8e0bd8fbe06d38a02f5c41fd26fb216bb8f2
2024-01-19 22:33:17     at /usr/local/cargo/git/checkouts/ethers-rs-3cecc26f7936b033/6c26645/ethers-providers/src/pending_transaction.rs:235
2024-01-19 22:33:17     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x440848ee03b252e5d6ba8c9fe68c68cc539676a01420272929b4c21ac4c90cc4, version: 3, nonce: 5122, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x0000000000000000000000004e2d04c65c399eb27b3e3ada06110bcd47b5a50600000000000000000000000000000000000000000000000000000000000031ae }, tx_gas_limit: Some(176104), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0da7c4b9c39f766a2df91b7351afedafd6b4ad7abffe1319fa81d386661b487fed000013e6d57b8612b31a05d378c6ee9f37726a8af89b75a7044b0c22c0bb6f30689f0fb46fe31cb2c6c0641e37209fc8bf0d8c82fd3f8a940d4148c40eea80417810a32c1b9da6e87b7a33f7e3c482596a30c1692936152e4f0227676584c1dba45df84e765ed321298222f0af4ac5c5b70d3ac4db5664e38b721124ce9c5a729189065c9c1c
2024-01-19 22:33:17     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 9, since_last_attempt_s: 22, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x440848ee03b252e5d6ba8c9fe68c68cc539676a01420272929b4c21ac4c90cc4, version: 3, nonce: 5122, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x0000000000000000000000004e2d04c65c399eb27b3e3ada06110bcd47b5a50600000000000000000000000000000000000000000000000000000000000031ae } }
2024-01-19 22:33:17     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-19 22:33:17 
2024-01-19 22:33:17   2024-01-20T06:33:17.550762Z  INFO hyperlane_ethereum::tx: confirmed transaction, tx_hash: 0xfb78a1dadd3818164737083ed88e8e0bd8fbe06d38a02f5c41fd26fb216bb8f2
2024-01-19 22:33:17     at chains/hyperlane-ethereum/src/tx.rs:52
2024-01-19 22:33:17     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x440848ee03b252e5d6ba8c9fe68c68cc539676a01420272929b4c21ac4c90cc4, version: 3, nonce: 5122, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x0000000000000000000000004e2d04c65c399eb27b3e3ada06110bcd47b5a50600000000000000000000000000000000000000000000000000000000000031ae }, tx_gas_limit: Some(176104), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0da7c4b9c39f766a2df91b7351afedafd6b4ad7abffe1319fa81d386661b487fed000013e6d57b8612b31a05d378c6ee9f37726a8af89b75a7044b0c22c0bb6f30689f0fb46fe31cb2c6c0641e37209fc8bf0d8c82fd3f8a940d4148c40eea80417810a32c1b9da6e87b7a33f7e3c482596a30c1692936152e4f0227676584c1dba45df84e765ed321298222f0af4ac5c5b70d3ac4db5664e38b721124ce9c5a729189065c9c1c
2024-01-19 22:33:17     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 9, since_last_attempt_s: 22, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x440848ee03b252e5d6ba8c9fe68c68cc539676a01420272929b4c21ac4c90cc4, version: 3, nonce: 5122, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x0000000000000000000000004e2d04c65c399eb27b3e3ada06110bcd47b5a50600000000000000000000000000000000000000000000000000000000000031ae } }
2024-01-19 22:33:17     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-19 22:33:17 
2024-01-19 22:33:17   2024-01-20T06:33:17.551478Z  INFO relayer::msg::pending_message: Message successfully processed by transaction, txid: 0x0000000000000000000000000000000000000000000000000000000000000000fb78a1dadd3818164737083ed88e8e0bd8fbe06d38a02f5c41fd26fb216bb8f2
2024-01-19 22:33:17     at agents/relayer/src/msg/pending_message.rs:243
2024-01-19 22:33:17     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 9, since_last_attempt_s: 22, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x440848ee03b252e5d6ba8c9fe68c68cc539676a01420272929b4c21ac4c90cc4, version: 3, nonce: 5122, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x0000000000000000000000004e2d04c65c399eb27b3e3ada06110bcd47b5a50600000000000000000000000000000000000000000000000000000000000031ae } }
2024-01-19 22:33:17     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-19 22:33:17 
2024-01-19 22:33:17   2024-01-20T06:33:17.901752Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118896..=5118900, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:17     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:17     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-19 22:33:17 
2024-01-19 22:33:18   2024-01-20T06:33:18.027367Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 26445..=27033, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:18     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:18     in hyperlane_base::contract_sync::ContractSync with label: "merkle_tree_hook", domain: "mevcommitsettlement"
2024-01-19 22:33:18 
2024-01-19 22:33:18   2024-01-20T06:33:18.028503Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 26445..=27033, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:18     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:18     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-19 22:33:18 
2024-01-19 22:33:18   2024-01-20T06:33:18.249432Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118896..=5118900, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:18     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:18     in hyperlane_base::contract_sync::ContractSync with label: "merkle_tree_hook", domain: "sepolia"
2024-01-19 22:33:18 
2024-01-19 22:33:48   2024-01-20T06:33:48.038956Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 27034..=27187, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:48     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:48     in hyperlane_base::contract_sync::ContractSync with label: "merkle_tree_hook", domain: "mevcommitsettlement"
2024-01-19 22:33:48 
2024-01-19 22:33:48   2024-01-20T06:33:48.041278Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 27034..=27187, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:48     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:48     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-19 22:33:48 
2024-01-19 22:33:48   2024-01-20T06:33:48.057716Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118901..=5118902, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:48     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-19 22:33:48     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-19 22:33:48 
2024-01-19 22:33:48   2024-01-20T06:33:48.490424Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5118901..=5118902, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-19 22:33:48     at hyperlane-base/src/contract_sync/mod.rs:71

Solution

Describe the solution you'd like

I'd like the agent binary built off main to allow the relayer to finalize transactions on our sidechain. This may involve changes in gas related tx params?

Describe alternatives you've considered

Is there any config I can use to change gas related tx params for the relayer?

Nice to Have

n/a

avious00 commented 8 months ago

hey @shaspitz just a heads up im subscribed to this issue and keeping tabs on our chats in tg/discord. let me know if the latest info Nam shared for debugging is helpful and what you find!

shaspitz commented 8 months ago

hey @shaspitz just a heads up im subscribed to this issue and keeping tabs on our chats in tg/discord. let me know if the latest info Nam shared for debugging is helpful and what you find!

Hey thanks for reaching out! I believe the info Nam shared in TG is in reference to a different issue related to messages not being relayed in FIFO ordering.

This issue is separate and has to do with the relayer being able to finalize transactions. Solving this issue will allow me to test/debug with the latest relayer source code

nambrot commented 8 months ago

Ah sorry, i misinterpreted this ticket. So sounds like you were able to pinpoint the issue to a forever pending tx in the mempool. Are you able to determine why the transaction is pending by any chance? gas price too low?

nambrot commented 8 months ago

Can you also confirm that spinning up the relayer again with the latest release does not fix the issue either?

shaspitz commented 8 months ago

No prob and I can confirm that when spinning up the relayer with this release, the issue exists. Just tried again and here's some more detailed logs where the tx is deemed timed out:

2024-01-23 21:38:27 
2024-01-23 21:38:28   2024-01-24T05:38:28.491475Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143662..=5143663, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:38:28     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:38:28     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:38:28 
2024-01-23 21:38:52   2024-01-24T05:38:52.124620Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143664..=5143665, num_logs: 1, estimated_time_to_sync: "synced"
2024-01-23 21:38:52     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:38:52     in hyperlane_base::contract_sync::ContractSync with label: "dispatched_messages", domain: "sepolia"
2024-01-23 21:38:52 
2024-01-23 21:38:52   2024-01-24T05:38:52.563497Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143664..=5143665, num_logs: 1, estimated_time_to_sync: "synced"
2024-01-23 21:38:52     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:38:52     in hyperlane_base::contract_sync::ContractSync with label: "merkle_tree_hook", domain: "sepolia"
2024-01-23 21:38:52 
2024-01-23 21:38:52   2024-01-24T05:38:52.710889Z DEBUG relayer::merkle_tree::builder: Ingesting leaf, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0
2024-01-23 21:38:52     at agents/relayer/src/merkle_tree/builder.rs:90
2024-01-23 21:38:52     in relayer::processor::main_loop with domain: sepolia
2024-01-23 21:38:52 
2024-01-23 21:38:52   2024-01-24T05:38:52.712294Z DEBUG relayer::msg::processor: Processor working on message, msg: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:38:52     at agents/relayer/src/msg/processor.rs:64
2024-01-23 21:38:52     in relayer::processor::main_loop with domain: sepolia
2024-01-23 21:38:52 
2024-01-23 21:38:52   2024-01-24T05:38:52.712627Z DEBUG relayer::msg::processor: Sending message to submitter, msg: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, nonce: 5152, .. }
2024-01-23 21:38:52     at agents/relayer/src/msg/processor.rs:95
2024-01-23 21:38:52     in relayer::processor::main_loop with domain: sepolia
2024-01-23 21:38:52 
2024-01-23 21:38:56   2024-01-24T05:38:56.975326Z DEBUG relayer::msg::metadata::multisig::base: Found checkpoint with quorum, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, metadata.checkpoint: CheckpointWithMessageId { checkpoint: Checkpoint { merkle_tree_hook_address: 0x0000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d, mailbox_domain: 11155111, root: 0x48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d4419, index: 5124 }, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0 }
2024-01-23 21:38:56     at agents/relayer/src/msg/metadata/multisig/base.rs:129
2024-01-23 21:38:56     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x0000000000000000000000000913412341c501441f80a493fe9cfb44b67f960a, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:56     in relayer::msg::metadata::aggregation::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:38:56     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:56     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:56     in relayer::msg::metadata::routing::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:38:56     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:56     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:56     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 0, since_last_attempt_s: 0, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:38:56     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:38:56 
2024-01-23 21:38:57   2024-01-24T05:38:57.553220Z DEBUG relayer::msg::metadata::multisig::base: Found checkpoint with quorum, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, metadata.checkpoint: CheckpointWithMessageId { checkpoint: Checkpoint { merkle_tree_hook_address: 0x0000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d, mailbox_domain: 11155111, root: 0x48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d4419, index: 5124 }, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0 }
2024-01-23 21:38:57     at agents/relayer/src/msg/metadata/multisig/base.rs:129
2024-01-23 21:38:57     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x0000000000000000000000008d9bb86d1176cf64bb81c9f1788437a79bd1c322, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:57     in relayer::msg::metadata::aggregation::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:38:57     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:57     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:57     in relayer::msg::metadata::routing::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:38:57     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:57     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:38:57     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 0, since_last_attempt_s: 0, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:38:57     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:38:57 
2024-01-23 21:38:57   2024-01-24T05:38:57.618077Z DEBUG relayer::msg::gas_payment: Evaluating if message meets gas payment requirement, msg: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, nonce: 5152, .. }, policy: GasPaymentPolicyNone, current_payment: InterchainGasPayment { message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, destination: 17864, payment: 0, gas_amount: 0 }, current_expenditure: InterchainGasExpenditure { message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, tokens_used: 0, gas_used: 0 }
2024-01-23 21:38:57     at agents/relayer/src/msg/gas_payment/mod.rs:109
2024-01-23 21:38:57     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 0, since_last_attempt_s: 0, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:38:57     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:38:57 
2024-01-23 21:38:57   2024-01-24T05:38:57.618451Z DEBUG relayer::msg::pending_message: Gas payment requirement met, ready to process message, gas_limit: 176080
2024-01-23 21:38:57     at agents/relayer/src/msg/pending_message.rs:192
2024-01-23 21:38:57     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 0, since_last_attempt_s: 0, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:38:57     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:38:57 
2024-01-23 21:38:57   2024-01-24T05:38:57.618801Z DEBUG relayer::msg::serial_submitter: Operation prepared, op: PendingMessage(PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } })
2024-01-23 21:38:57     at agents/relayer/src/msg/serial_submitter.rs:179
2024-01-23 21:38:57     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:38:57 
2024-01-23 21:38:57   2024-01-24T05:38:57.622421Z  INFO hyperlane_ethereum::tx: Dispatching transaction, to: 0x4426b0196dc89c67021a81872bcffaf6a4d8486c, data: 0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000
2024-01-23 21:38:57     at chains/hyperlane-ethereum/src/tx.rs:38
2024-01-23 21:38:57     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:38:57     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:38:57     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:38:57 
2024-01-23 21:38:57   2024-01-24T05:38:57.636230Z  INFO hyperlane_ethereum::tx: Dispatched tx, to: 0x4426b0196dc89c67021a81872bcffaf6a4d8486c, data: 0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000, tx_hash: 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412
2024-01-23 21:38:57     at chains/hyperlane-ethereum/src/tx.rs:45
2024-01-23 21:38:57     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:38:57     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:38:57     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:38:57 
2024-01-23 21:38:57   2024-01-24T05:38:57.693814Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 65347..=65501, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:38:57     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:38:57     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:38:57 
2024-01-23 21:38:58   2024-01-24T05:38:58.579922Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143664..=5143665, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:38:58     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:38:58     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:38:58 
2024-01-23 21:39:04   2024-01-24T05:39:04.638172Z DEBUG ethers_providers::pending_transaction: Starting to poll pending tx 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412
2024-01-23 21:39:04     at /usr/local/cargo/git/checkouts/ethers-rs-3cecc26f7936b033/c9ced03/ethers-providers/src/pending_transaction.rs:175
2024-01-23 21:39:04     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:39:04     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:39:04     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:39:04 
2024-01-23 21:39:27   2024-01-24T05:39:27.700777Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 65502..=65654, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:39:27     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:39:27     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:39:27 
2024-01-23 21:39:28   2024-01-24T05:39:28.675823Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143666..=5143668, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:39:28     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:39:28     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:39:28 
2024-01-23 21:39:57   2024-01-24T05:39:57.716736Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 65655..=65808, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:39:57     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:39:57     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:39:57 
2024-01-23 21:39:58   2024-01-24T05:39:58.774689Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143669..=5143669, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:39:58     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:39:58     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:39:58 
2024-01-23 21:40:27   2024-01-24T05:40:27.724484Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 65809..=65959, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:40:27     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:40:27     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:40:27 
2024-01-23 21:40:28   2024-01-24T05:40:28.967551Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143670..=5143670, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:40:28     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:40:28     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:40:28 
2024-01-23 21:40:57   2024-01-24T05:40:57.726322Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 65960..=66110, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:40:57     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:40:57     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:40:57 
2024-01-23 21:40:59   2024-01-24T05:40:59.049209Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143671..=5143672, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:40:59     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:40:59     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:40:59 
2024-01-23 21:41:27   2024-01-24T05:41:27.721053Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 66111..=66263, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:41:27     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:41:27     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:41:27 
2024-01-23 21:41:29   2024-01-24T05:41:29.127928Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143673..=5143675, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:41:29     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:41:29     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:41:29 
2024-01-23 21:41:57   2024-01-24T05:41:57.739300Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 66264..=66416, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:41:57     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:41:57     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:41:57 
2024-01-23 21:41:59   2024-01-24T05:41:59.239658Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143676..=5143677, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:41:59     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:41:59     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:41:59 
2024-01-23 21:42:27   2024-01-24T05:42:27.748854Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 66417..=66567, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:42:27     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:42:27     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:42:27 
2024-01-23 21:42:29   2024-01-24T05:42:29.427759Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143678..=5143680, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:42:29     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:42:29     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:42:29 
2024-01-23 21:42:57   2024-01-24T05:42:57.757672Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 66568..=66718, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:42:57     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:42:57     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:42:57 
2024-01-23 21:42:59   2024-01-24T05:42:59.518096Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143681..=5143682, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:42:59     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:42:59     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:42:59 
2024-01-23 21:43:27   2024-01-24T05:43:27.761150Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 66719..=66870, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:43:27     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:43:27     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:43:27 
2024-01-23 21:43:29   2024-01-24T05:43:29.621419Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143683..=5143685, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:43:29     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:43:29     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:43:29 
2024-01-23 21:43:57   2024-01-24T05:43:57.625481Z ERROR hyperlane_ethereum::tx: waiting for receipt timed out, tx_hash: 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412, error: Elapsed(())
2024-01-23 21:43:57     at chains/hyperlane-ethereum/src/tx.rs:63
2024-01-23 21:43:57     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:43:57     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:43:57     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:43:57 
2024-01-23 21:43:57   2024-01-24T05:43:57.629172Z  WARN relayer::msg::pending_message: Error when processing message, error: TransactionTimeout
2024-01-23 21:43:57     at agents/relayer/src/msg/pending_message.rs:233
2024-01-23 21:43:57     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 0, since_last_attempt_s: 4, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:43:57     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:43:57 
2024-01-23 21:43:57   2024-01-24T05:43:57.768029Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 66871..=67021, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:43:57     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:43:57     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:43:57 
2024-01-23 21:43:59   2024-01-24T05:43:59.707074Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143686..=5143687, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:43:59     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:43:59     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"
2024-01-23 21:43:59 
2024-01-23 21:44:11   2024-01-24T05:44:11.824683Z DEBUG relayer::msg::metadata::multisig::base: Found checkpoint with quorum, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, metadata.checkpoint: CheckpointWithMessageId { checkpoint: Checkpoint { merkle_tree_hook_address: 0x0000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d, mailbox_domain: 11155111, root: 0x48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d4419, index: 5124 }, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0 }
2024-01-23 21:44:11     at agents/relayer/src/msg/metadata/multisig/base.rs:129
2024-01-23 21:44:11     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x0000000000000000000000000913412341c501441f80a493fe9cfb44b67f960a, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:11     in relayer::msg::metadata::aggregation::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:11     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:11     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:11     in relayer::msg::metadata::routing::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:11     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:11     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:11     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 1, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:11     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:11 
2024-01-23 21:44:12   2024-01-24T05:44:12.663231Z DEBUG relayer::msg::metadata::multisig::base: Found checkpoint with quorum, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, metadata.checkpoint: CheckpointWithMessageId { checkpoint: Checkpoint { merkle_tree_hook_address: 0x0000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d, mailbox_domain: 11155111, root: 0x48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d4419, index: 5124 }, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0 }
2024-01-23 21:44:12     at agents/relayer/src/msg/metadata/multisig/base.rs:129
2024-01-23 21:44:12     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x0000000000000000000000008d9bb86d1176cf64bb81c9f1788437a79bd1c322, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:12     in relayer::msg::metadata::aggregation::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:12     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:12     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:12     in relayer::msg::metadata::routing::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:12     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:12     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:12     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 1, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:12     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:12   2024-01-24T05:44:12.722608Z DEBUG relayer::msg::gas_payment: Evaluating if message meets gas payment requirement, msg: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, nonce: 5152, .. }, policy: GasPaymentPolicyNone, current_payment: InterchainGasPayment { message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, destination: 17864, payment: 0, gas_amount: 0 }, current_expenditure: InterchainGasExpenditure { message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, tokens_used: 0, gas_used: 0 }
2024-01-23 21:44:12     at agents/relayer/src/msg/gas_payment/mod.rs:109
2024-01-23 21:44:12     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 1, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:12     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:12   2024-01-24T05:44:12.722779Z DEBUG relayer::msg::pending_message: Gas payment requirement met, ready to process message, gas_limit: 176080
2024-01-23 21:44:12     at agents/relayer/src/msg/pending_message.rs:192
2024-01-23 21:44:12     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 1, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:12     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:12   2024-01-24T05:44:12.722871Z DEBUG relayer::msg::serial_submitter: Operation prepared, op: PendingMessage(PendingMessage { num_retries: 1, since_last_attempt_s: 15, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } })
2024-01-23 21:44:12     at agents/relayer/src/msg/serial_submitter.rs:179
2024-01-23 21:44:12     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:12   2024-01-24T05:44:12.728419Z  INFO hyperlane_ethereum::tx: Dispatching transaction, to: 0x4426b0196dc89c67021a81872bcffaf6a4d8486c, data: 0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000
2024-01-23 21:44:12     at chains/hyperlane-ethereum/src/tx.rs:38
2024-01-23 21:44:12     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:44:12     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 1, since_last_attempt_s: 15, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:12     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:12   2024-01-24T05:44:12.733224Z  WARN hyperlane_ethereum::rpc_clients: Non-retryable JsonRpcError in http provider, error: (code: -32000, message: already known, data: None)
2024-01-23 21:44:12     at chains/hyperlane-ethereum/src/rpc_clients/mod.rs:87
2024-01-23 21:44:12     in hyperlane_ethereum::rpc_clients::fallback::request_with_fallback with fallback_count: 0, provider_index: 0, provider: PrometheusJsonRpcClient(Http { id: 3938, url: http://172.29.0.98:8545/ })
2024-01-23 21:44:12     in hyperlane_ethereum::rpc_clients::fallback::request with self: FallbackProvider { chain_name: "mevcommitsettlement", hosts: "172.29.0.98:8545" }, method: "eth_sendRawTransaction", params: [String("0x02f902738245c8068405f5e1008405f5e10e8302afd0944426b0196dc89c67021a81872bcffaf6a4d8486c80b902047c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000c001a0c1d316c7630f594ec88196ee84f70fafb4304294b0025f355fed8a567f0673d5a079fe4f198d005f8a279fe01d8428b9b6b140ec23a8ff4863d5f880f252233443")]
2024-01-23 21:44:12     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:44:12     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 1, since_last_attempt_s: 15, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:12     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:12   2024-01-24T05:44:12.733852Z  WARN relayer::msg::pending_message: Error when processing message, error: ContractError(MiddlewareError(MiddlewareError(MiddlewareError(MiddlewareError(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32000, message: "already known", data: None })))))))
2024-01-23 21:44:12     at agents/relayer/src/msg/pending_message.rs:233
2024-01-23 21:44:12     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 1, since_last_attempt_s: 15, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:12     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:44:12 
2024-01-23 21:44:27   2024-01-24T05:44:27.083404Z DEBUG relayer::msg::metadata::multisig::base: Found checkpoint with quorum, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, metadata.checkpoint: CheckpointWithMessageId { checkpoint: Checkpoint { merkle_tree_hook_address: 0x0000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d, mailbox_domain: 11155111, root: 0x48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d4419, index: 5124 }, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0 }
2024-01-23 21:44:27     at agents/relayer/src/msg/metadata/multisig/base.rs:129
2024-01-23 21:44:27     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x0000000000000000000000000913412341c501441f80a493fe9cfb44b67f960a, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::aggregation::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:27     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::routing::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:27     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 2, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.662201Z DEBUG relayer::msg::metadata::multisig::base: Found checkpoint with quorum, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, metadata.checkpoint: CheckpointWithMessageId { checkpoint: Checkpoint { merkle_tree_hook_address: 0x0000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d, mailbox_domain: 11155111, root: 0x48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d4419, index: 5124 }, message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0 }
2024-01-23 21:44:27     at agents/relayer/src/msg/metadata/multisig/base.rs:129
2024-01-23 21:44:27     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x0000000000000000000000008d9bb86d1176cf64bb81c9f1788437a79bd1c322, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::aggregation::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:27     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000bfe44816617c070e2cbb7b753aab96a5993a6db7, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::routing::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }
2024-01-23 21:44:27     in relayer::msg::metadata::base::build_ism_and_metadata with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::metadata::base::build with ism_address: 0x000000000000000000000000a48170e24a7ddf149a35274138a7ccbe0b69ac72, HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, domain: "mevcommitsettlement"
2024-01-23 21:44:27     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 2, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.724388Z DEBUG relayer::msg::gas_payment: Evaluating if message meets gas payment requirement, msg: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, nonce: 5152, .. }, policy: GasPaymentPolicyNone, current_payment: InterchainGasPayment { message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, destination: 17864, payment: 0, gas_amount: 0 }, current_expenditure: InterchainGasExpenditure { message_id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, tokens_used: 0, gas_used: 0 }
2024-01-23 21:44:27     at agents/relayer/src/msg/gas_payment/mod.rs:109
2024-01-23 21:44:27     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 2, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.724553Z DEBUG relayer::msg::pending_message: Gas payment requirement met, ready to process message, gas_limit: 176080
2024-01-23 21:44:27     at agents/relayer/src/msg/pending_message.rs:192
2024-01-23 21:44:27     in relayer::msg::pending_message::prepare with self: PendingMessage { num_retries: 2, since_last_attempt_s: 10, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.724641Z DEBUG relayer::msg::serial_submitter: Operation prepared, op: PendingMessage(PendingMessage { num_retries: 2, since_last_attempt_s: 14, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } })
2024-01-23 21:44:27     at agents/relayer/src/msg/serial_submitter.rs:179
2024-01-23 21:44:27     in relayer::msg::serial_submitter::prepare_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.727231Z  INFO hyperlane_ethereum::tx: Dispatching transaction, to: 0x4426b0196dc89c67021a81872bcffaf6a4d8486c, data: 0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000
2024-01-23 21:44:27     at chains/hyperlane-ethereum/src/tx.rs:38
2024-01-23 21:44:27     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:44:27     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 2, since_last_attempt_s: 14, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.729806Z  WARN hyperlane_ethereum::rpc_clients: Non-retryable JsonRpcError in http provider, error: (code: -32000, message: already known, data: None)
2024-01-23 21:44:27     at chains/hyperlane-ethereum/src/rpc_clients/mod.rs:87
2024-01-23 21:44:27     in hyperlane_ethereum::rpc_clients::fallback::request_with_fallback with fallback_count: 0, provider_index: 0, provider: PrometheusJsonRpcClient(Http { id: 3950, url: http://172.29.0.98:8545/ })
2024-01-23 21:44:27     in hyperlane_ethereum::rpc_clients::fallback::request with self: FallbackProvider { chain_name: "mevcommitsettlement", hosts: "172.29.0.98:8545" }, method: "eth_sendRawTransaction", params: [String("0x02f902738245c8068405f5e1008405f5e10e8302afd0944426b0196dc89c67021a81872bcffaf6a4d8486c80b902047c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000c001a0c1d316c7630f594ec88196ee84f70fafb4304294b0025f355fed8a567f0673d5a079fe4f198d005f8a279fe01d8428b9b6b140ec23a8ff4863d5f880f252233443")]
2024-01-23 21:44:27     in hyperlane_ethereum::mailbox::process with HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d }, tx_gas_limit: Some(176080), metadata: 0x00000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c
2024-01-23 21:44:27     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 2, since_last_attempt_s: 14, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.729935Z  WARN relayer::msg::pending_message: Error when processing message, error: ContractError(MiddlewareError(MiddlewareError(MiddlewareError(MiddlewareError(JsonRpcClientError(JsonRpcError(JsonRpcError { code: -32000, message: "already known", data: None })))))))
2024-01-23 21:44:27     at agents/relayer/src/msg/pending_message.rs:233
2024-01-23 21:44:27     in relayer::msg::pending_message::submit with self: PendingMessage { num_retries: 2, since_last_attempt_s: 14, next_attempt_after_s: 0, message: HyperlaneMessage { id: 0x1f9fe99ba028e5e573f3b4bc94332f52c01f51023ba986954c48f1df040b8cf0, version: 3, nonce: 5152, origin: sepolia, sender: 0x6bc317f62fc73c146f7016468fc9a1c60f34e6c7, destination: 17864, recipient: 0x000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc, body: 0x00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d } }
2024-01-23 21:44:27     in relayer::msg::serial_submitter::submit_task with domain: mevcommitsettlement
2024-01-23 21:44:27 
2024-01-23 21:44:27   2024-01-24T05:44:27.773025Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 67022..=67176, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:44:27     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:44:27     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "mevcommitsettlement"
2024-01-23 21:44:27 
2024-01-23 21:44:29   2024-01-24T05:44:29.901060Z  INFO hyperlane_base::contract_sync: Found log(s) in index range, range: 5143688..=5143690, num_logs: 0, estimated_time_to_sync: "synced"
2024-01-23 21:44:29     at hyperlane-base/src/contract_sync/mod.rs:71
2024-01-23 21:44:29     in hyperlane_base::contract_sync::ContractSync with label: "gas_payments", domain: "sepolia"

shaspitz commented 8 months ago

Gas parameters are a possible culprit here 👍, in our golang client we set nonce explicitly, as well as the value for gasPrice via this method and priority fee via this method.

Is there any PR merged between Nov 20th and Dec 14th 2023 (timeframe between working and non-working relayer binary) that changed gas parameters or how the ethers client is used?

shaspitz commented 8 months ago

Also here's the output of cast tx for the tx that failed just now if that helps

cast tx 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412

blockHash
blockNumber
from                 0x0DCaa27B9E4Db92F820189345792f8eC5Ef148F6
gas                  176080
gasPrice             100000014
hash                 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412
input                0x7c39d1300000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000000000000000000000000000000000000014000000000000000000000000000000000000000000000000000000000000000d600000010000000d600000000000000000000000000000000000000004917a9746a7b6e0a57159ccb7f5a6744247f2d0d48dc67431af9df5cb117c0ac83a93308cb0546200d4f0099dc6789033f4d441900001404b239e39c2466de89f424455f3fe2e210747daf5e5d93a3bce1fe325bbcf3441d542fd9954d7606d9862a1c8167cfd8601e612f5d7b2a8fc5ea3de1d5ea9927c81b52ece2ca903a0065b76255dd9c53a4a8441daf76c96ea40752b0d1a66fb06b4d4de5a3ad52b915c104ad5173297c100fd0ba9fa3da70d42c3572da16f0e7a71c1c00000000000000000000000000000000000000000000000000000000000000000000000000000000008d030000142000aa36a70000000000000000000000006bc317f62fc73c146f7016468fc9a1c60f34e6c7000045c8000000000000000000000000bd610046d6e6b94f971fdf5d06800e74c3d81ccc00000000000000000000000004f713a0b687c84d4f66acd1423712af6f852b78000000000000000000000000000000000000000000000000000000000000064d00000000000000000000000000000000000000
nonce                6
r                    0xc1d316c7630f594ec88196ee84f70fafb4304294b0025f355fed8a567f0673d5
s                    0x79fe4f198d005f8a279fe01d8428b9b6b140ec23a8ff4863d5f880f252233443
to                   0x4426B0196DC89C67021A81872BcFFAf6A4D8486C
transactionIndex
v                    1
value                0
yParity              1

nambrot commented 8 months ago

When you say failed just now you mean this error? { code: -32000, message: "already known", data: None }

I believe that error gets thrown if the exact transaction is already in the mempool. So my question would be is why is this transaction not included? Is the gasPrice too low? Because the relayer retries and fetches the relevant information fresh on every try IMO

shaspitz commented 8 months ago

Sorry, "failed" was not the right word there, I meant that tx with hash 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412 is indeed staying pending when I tested again. It looks like the first error is ERROR hyperlane_ethereum::tx: waiting for receipt timed out, tx_hash: 0x68c4d92e2277f4a51c94e6dc32c2649454040d6f00dc05cf356cad5507033412

why is this transaction not included

My best guess is that yes the gas price or tip is being set too low. In original post I was able to cancel the pending tx by sending a new transaction with the same nonce and higher gas price/tip

Since we see the already known error, this means the exact same tx is being submitted to mempool multiple times. Whereas if tx retries involved increasing gasPrice, the retry tx would have a different hash and would not give that error

shaspitz commented 8 months ago

It's also possible the relayer is obtaining incorrect gas price/tip suggestions, rpc methods mentioned above have worked for other clients we use

nambrot commented 8 months ago

I think it would still be useful to know why the tx, even when resubmitted, is not using the right values. It persisting to me would indicate that the relayer is indeed somehow obtaining or receiving the wrong suggestions for both which would be important to dig into

nambrot commented 8 months ago

The closest change I can think of we made was this one https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/2986, which is related to https://github.com/hyperlane-xyz/ethers-rs/pull/10 and https://github.com/hyperlane-xyz/ethers-rs/pull/8

nambrot commented 8 months ago

It looks like the txs submitted are not type 2 (eip1559) tho?

avious00 commented 8 months ago

hey @shaspitz wanted to check if this was resolved here or on tg

shaspitz commented 8 months ago

Sorry for late response yall, here's an example of client side tx configuration that has worked for us.

What's notable is that we submit eip1559 type transactions. The value for gasTip is set explicitly and comes from eth_maxPriorityFeePerGas rpc method, the value for gasFeeCap is the value returned from eth_gasPrice rpc method + gasTip value.

I just tested, printing the config before a tx is sent, and some concrete values for these parameters are as follow:

GasPrice:<nil> GasFeeCap:+2000000007 GasTipCap:+1000000000 GasLimit:3000000

I don't have much context on exactly how your rust client works, but the two PRs you linked seem like good guesses as to what is causing this issue. Notably the hardcoded value from https://github.com/hyperlane-xyz/ethers-rs/pull/8 seems smaller than what's successfully used by our client-side code

nambrot commented 8 months ago

You can see the default 1559 logic from ethers-rs here https://github.com/hyperlane-xyz/ethers-rs/blob/733c8378d3c320a5550b93cd0c19b88d1758433a/ethers-core/src/utils/mod.rs#L436

For the gas price config that was used there, can you tell why its not being included? According to our understanding, the default estimation logic should work for 1559 spec standard, so only when chains deviate from that I would expect non-inclusion (BSC for example has a hard coded minimum tip). So right now my hunch would be that your chain maybe similarly has constraints that are not obvious deviations from the spec. I see above that the minimum tip has been used?

shaspitz commented 8 months ago

So these value GasPrice:<nil> GasFeeCap:+2000000007 GasTipCap:+1000000000 GasLimit:3000000 are from the golang client we use, and those parameters allow txes to successfully be included in a block.

w.r.t the default eip 1559 estimation logic you linked, this PR is a deviation from that logic, correct? Notably, the logic that existed prior to the Nov 29th PRs works as expected to finalize txes on our chain.

As far as I know our chain follows standard eip 1559.

Do you know why this change was made https://github.com/hyperlane-xyz/ethers-rs/pull/8? It seems directly relevant to the issue we're discussing, in that default gasTip is now lower. 100_000_000 vs the value that works from our golang client, 1_000_000_000

nambrot commented 8 months ago

There was a hard coded 3 gwei minimum tip which is much higher than necessary. Most chains, especially rollups, do not have that high of a required tip, so this was essentially forcing our relayer to overpay 30x as the tip. I understand that with 1 gwei of a priority tip, you could get transactions included, but I would look into why 0.1 gwei is not enough. As far as I can tell, the logic uses the fee_history RPC url to assess a reasonable fee and lower bounds it by 0.1 gwei.

Alternatives are to add a feature to include overrides at the agent level for gas parameters from configuration (we have that on the TS side), or create a fork that sets a higher lower bound that you maintain until we have the gas parameter configuration in main

shaspitz commented 8 months ago

w.r.t gas params on our chain, we don't do anything out of the ordinary. Full geth params available here: https://github.com/ethereum/go-ethereum/blob/master/cmd/utils/flags.go

Summary of our config:

We use the default geth param for "minimum gas price for mining a transaction", this is 1 gwei -> geth src
We use the default geth param for "maximum transaction priority fee (or gasprice before London fork) to be recommended by gpo", this is 500 gwei -> geth src
We use default geth param for "gas price below which gpo will ignore transactions", this is 2 wei -> geth src
There does not seem to be a parameter for minimum tip other than what's mentioned here

We have the freedom to change those parameters, but given most evm chains use a fork of geth, it seems reasonable to expect the relayer to function with default geth values. Also see: https://etherscan.io/gastracker.

w.r.t eth_feeHistory RPC, the REWARD object containing priority fees is optional. Ie. feeHistory is not guaranteed to give proper estimate of priority fee, esp if there are empty blocks. eth_maxPriorityFeePerGas seems preferable in that it returns a direct estimate

shaspitz commented 8 months ago

Update: geth's miner.gasprice flag is akin to a minimum priority fee enforced by nodes, default is indeed 1 gwei. For spam prevention purposes we would like to keep this param at or above 1 gwei. Ideally the relayer would be able to obtain an estimate for this value via eth_maxPriorityFeePerGas, and still be compatible with other chains that have lower min priority fees

nambrot commented 8 months ago

Thanks for the thorough investigation! I agree that the usage of eth_maxPriorityFeePerGas would be a good idea and not rely on just the feeHistory RPC. I found this article quite helpful https://docs.alchemy.com/docs/how-to-build-a-gas-fee-estimator-using-eip-1559. Would definitely support a PR if you feel up for it! If not, we'll have to see if we can get it prioritized on our end, cc @avious00.

Fwiw, plenty of chains have <1 gwei gas prices, I would say the default in geth is probably outdated.

https://owlracle.info/opt https://owlracle.info/arb https://owlracle.info/base

shaspitz commented 8 months ago

Valuable info! Rn our team is focused on getting a working bridge up for our chain, so I've been building a temporary solution. Happy to circle back and make a PR on this issue in the future

tkporter commented 4 months ago

Following up here - we have a workaround merged that allows for agent transaction overrides to be set: https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3563

You can manually configure the transactionOverrides as we do here https://github.com/hyperlane-xyz/hyperlane-monorepo/pull/3563/files#diff-6287a71fca6adf8b4b4e85011e0a3f122b202810e876a5d38f7251c6a0c4874aR921. I'm not certain if this is included in the latest public release, but if you use this image gcr.io/abacus-labs-dev/hyperlane-agent:37af3dc-20240522-170404 you can make use of this.

@nambrot do you think this covers this issue or it's worth keeping this around for alternative fee calculation logic?

nambrot commented 4 months ago

I think this actually is a pretty good workaround, I would let @shaspitz chime in on whether that works for them

hyperlane-xyz / hyperlane-monorepo