search

bnb-chain / op-geth

GNU Lesser General Public License v3.0
54 stars 44 forks source link

op-node's `pre fetch receipts fail without error,need retry` and op-geth's `Peer's transaction rejected` logs #73

Closed e1ghtball closed 3 months ago

e1ghtball commented 6 months ago

System information

Network: mainnet

if you are running a local node, please provide the following information: op-node version: op-node version v0.0.0-8f623e64-1705481865 op-geth version: op-geth version 0.1.0-unstable-77340c47 OS & Version: Rocky Linux 9.2

Expected behaviour

op-node: regular Optimistically queueing unsafe L2 execution payload/Sync progress/Received signed execution payload from p2p logs op-geth regular Chain head was updated/Imported new potential chain segment logs

Actual behaviour

op-node: A lot of pre fetch receipts fail without error,need retry logs with the same block numbers:

10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xa756e46bf5e097478b212f9718b8ebf073f6a779ca4a77c2ee787aaf522e346b blockNumber=36,547,726
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfa4762b9b0a501a89f80cf9afb8df4d2a26419b1c3a739a7ea9ac6565a34c9de blockNumber=36,547,731
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x2de052cd55ba784ab6b11e60249d72d0f6608769ebd0ae3afc32e420bb6f5776 blockNumber=36,547,723
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xaa1da3bc77bfcf57046c7b0639fa5b5f2ce90c921edd1afa3f9a19f758afdf02 blockNumber=36,547,724
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xe34905997269807854f1814f34204b1a15ca334603721d4b9d678e0f173f9eef blockNumber=36,547,729
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x1f3050ef986f638e8db648c84115870ec9b2bba306d8ed890a5667fb363e0f68 blockNumber=36,547,730
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="L1 head moved forward"                  l1_head=0x23aad59936fcb01603a879772e089b435282f8390c95dd0c6accb523e7fc5406:36948154
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=153
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="validated block"                        result=ACCEPT from=16Uiu2HAmTJa7xJJYpK7m7ynBe2TcSy5es7BeTn88WXydsXg4qikK
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=info msg="Received signed execution payload from p2p" id=0x8b1b8cee1b04b65df017847c67460095a91b132184ecc1817bd22ceb97d76877:18624578 peer=16Uiu2HAmTJa7xJJYpK7m7ynBe2TcSy5es7BeTn88WXydsXg4qikK
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x8b1b8cee1b04b65df017847c67460095a91b132184ecc1817bd22ceb97d76877:18624578
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=153
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xd6e14a49b604a1c59478c3488fed0a029474241b8489e238b172d48a20d4d7e6 blockNumber=36,547,722
10|opbnb-node  | t=2024-03-14T01:05:01+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x9d653736f18a9b0d6045e2f1f4ac3fdf14576b80258fed2c44e0edda6da9a797 blockNumber=36,547,725
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x3cb029bb832889438772af96c82b048b785dc0a75cd5b35bbe129340e3409694 blockNumber=36,547,728
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="validated block"                        result=ACCEPT from=16Uiu2HAmPm56nzE7iPBHCV3NkFcbCH8PR7c26BeG8kwbv1ajCyiF
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=info msg="Received signed execution payload from p2p" id=0x6f9af6dc76ab7e864d90814adf4454adbc92ebc0d3fd5889ca33ace2df2d3184:18624579 peer=16Uiu2HAmPm56nzE7iPBHCV3NkFcbCH8PR7c26BeG8kwbv1ajCyiF
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x6f9af6dc76ab7e864d90814adf4454adbc92ebc0d3fd5889ca33ace2df2d3184:18624579
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=153
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xa756e46bf5e097478b212f9718b8ebf073f6a779ca4a77c2ee787aaf522e346b blockNumber=36,547,726
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xa1eaeb37598cb27432d24bb9167946ea5439f296c8dc1a29732fa717ad482552 blockNumber=36,547,727
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfa4762b9b0a501a89f80cf9afb8df4d2a26419b1c3a739a7ea9ac6565a34c9de blockNumber=36,547,731
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="requesting missing unsafe L2 block range" start=0x2aecf2269ea72a5c278fc835e17b6d69bb8331775dc11076e694b3ca97daa30a:18070245 end=0x3de4315f2b57d7c9211085fce4fbd58fc8e01f62641854dbbec9456d01779bd2:18622305 size=552,060
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="ignoring request to sync L2 range, timestamp is too old for p2p" start=0x2aecf2269ea72a5c278fc835e17b6d69bb8331775dc11076e694b3ca97daa30a:18070245 end=0x3de4315f2b57d7c9211085fce4fbd58fc8e01f62641854dbbec9456d01779bd2:18622305 start_time=1,709,823,968
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x2de052cd55ba784ab6b11e60249d72d0f6608769ebd0ae3afc32e420bb6f5776 blockNumber=36,547,723
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xaa1da3bc77bfcf57046c7b0639fa5b5f2ce90c921edd1afa3f9a19f758afdf02 blockNumber=36,547,724
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xe34905997269807854f1814f34204b1a15ca334603721d4b9d678e0f173f9eef blockNumber=36,547,729
10|opbnb-node  | t=2024-03-14T01:05:02+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x1f3050ef986f638e8db648c84115870ec9b2bba306d8ed890a5667fb363e0f68 blockNumber=36,547,730
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xd6e14a49b604a1c59478c3488fed0a029474241b8489e238b172d48a20d4d7e6 blockNumber=36,547,722
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x9d653736f18a9b0d6045e2f1f4ac3fdf14576b80258fed2c44e0edda6da9a797 blockNumber=36,547,725
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x3cb029bb832889438772af96c82b048b785dc0a75cd5b35bbe129340e3409694 blockNumber=36,547,728
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="validated block"                        result=ACCEPT from=16Uiu2HAmGbJGCngYXM3FJ7wUydrS6tuE9kpwvmfgdNZKfmP6dJmf
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=info msg="Received signed execution payload from p2p" id=0xe327e8e911e4715481bf497820015c7fb6c7f83f42eaa1e2232d8dd9d648fc2b:18624580 peer=16Uiu2HAmGbJGCngYXM3FJ7wUydrS6tuE9kpwvmfgdNZKfmP6dJmf
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xe327e8e911e4715481bf497820015c7fb6c7f83f42eaa1e2232d8dd9d648fc2b:18624580
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=153
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfa4762b9b0a501a89f80cf9afb8df4d2a26419b1c3a739a7ea9ac6565a34c9de blockNumber=36,547,731
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xa756e46bf5e097478b212f9718b8ebf073f6a779ca4a77c2ee787aaf522e346b blockNumber=36,547,726
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xa1eaeb37598cb27432d24bb9167946ea5439f296c8dc1a29732fa717ad482552 blockNumber=36,547,727
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x2de052cd55ba784ab6b11e60249d72d0f6608769ebd0ae3afc32e420bb6f5776 blockNumber=36,547,723
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xaa1da3bc77bfcf57046c7b0639fa5b5f2ce90c921edd1afa3f9a19f758afdf02 blockNumber=36,547,724
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xe34905997269807854f1814f34204b1a15ca334603721d4b9d678e0f173f9eef blockNumber=36,547,729
10|opbnb-node  | t=2024-03-14T01:05:03+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x1f3050ef986f638e8db648c84115870ec9b2bba306d8ed890a5667fb363e0f68 blockNumber=36,547,730
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xd6e14a49b604a1c59478c3488fed0a029474241b8489e238b172d48a20d4d7e6 blockNumber=36,547,722
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x9d653736f18a9b0d6045e2f1f4ac3fdf14576b80258fed2c44e0edda6da9a797 blockNumber=36,547,725
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="Revalidated node"                       p2p=discv5 b=14 id=d6594bc1d339110c5b63c6dcbd26f0d89ecdd1dcd5bd0ff9a119a9f852abf137 checks=45
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="validated block"                        result=ACCEPT from=16Uiu2HAmGbJGCngYXM3FJ7wUydrS6tuE9kpwvmfgdNZKfmP6dJmf
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=info msg="Received signed execution payload from p2p" id=0xa6f2855bedd2ea4a2ec7c06c95d2c407d24afefccc2474e3f9f1596fab19f7c7:18624581 peer=16Uiu2HAmGbJGCngYXM3FJ7wUydrS6tuE9kpwvmfgdNZKfmP6dJmf
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xa6f2855bedd2ea4a2ec7c06c95d2c407d24afefccc2474e3f9f1596fab19f7c7:18624581
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=153
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x3cb029bb832889438772af96c82b048b785dc0a75cd5b35bbe129340e3409694 blockNumber=36,547,728
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="requesting missing unsafe L2 block range" start=0x2aecf2269ea72a5c278fc835e17b6d69bb8331775dc11076e694b3ca97daa30a:18070245 end=0x3de4315f2b57d7c9211085fce4fbd58fc8e01f62641854dbbec9456d01779bd2:18622305 size=552,060
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="ignoring request to sync L2 range, timestamp is too old for p2p" start=0x2aecf2269ea72a5c278fc835e17b6d69bb8331775dc11076e694b3ca97daa30a:18070245 end=0x3de4315f2b57d7c9211085fce4fbd58fc8e01f62641854dbbec9456d01779bd2:18622305 start_time=1,709,823,968
10|opbnb-node  | t=2024-03-14T01:05:04+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xa756e46bf5e097478b212f9718b8ebf073f6a779ca4a77c2ee787aaf522e346b blockNumber=36,547,726

op-geth: Peer's transaction rejected logs:

11|opbnb-geth  | WARN [03-14|01:02:37.886] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0xce21b8e0633fd97b8b8de262db0e35b841f3d29106da7faafe4999b462c47d73 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:37.894] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0x3df7f03751c5751537989e3f6ac232e4af0fa877b04dd35aacaf668688650cb1 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:37.987] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0x014b3f85ff106580ae48c34787091d3a6e42e651afe8bcda95d00e41436d03fb err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.039] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0xb0af5495b985fe9a13425b434b602a321c25731ae7e7c888f7bc17dd0ccf1642 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.174] Peer's transaction rejected              peer=d6c86efc6657cb027ab912fa53e6188a893991ef6b2ec50c43a93e79e6b0978e txHash=0xbed2b8915f1bc18e1dd1e13e2d653c24d3a42e275812a615a4fbc2205794aa6c err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.211] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0x0b907de9807fed411f588fb4b2d1dbf96b7b9d6a867c949a9b2db54ade8f1b38 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.236] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0x04ee3b700505b62e437d3b8d2ff6cee1cdb8126ca86998057ff0b88d941260c4 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.511] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0xd638eddbdd789b42f075a2ca2bc96de021a58b176b1e579cecf598c0aee6c349 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.524] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0x014b3f85ff106580ae48c34787091d3a6e42e651afe8bcda95d00e41436d03fb err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.527] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0xb165622652a003b2c2ba37c21c1af5f44f8a7ca0588d94fbb58242ae788daa27 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.577] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0xa98f3736a00659df972453a6b222c5031bade0fef7a6ea5d85daded80b492bb0 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.584] Peer's transaction rejected              peer=d6c86efc6657cb027ab912fa53e6188a893991ef6b2ec50c43a93e79e6b0978e txHash=0xb165622652a003b2c2ba37c21c1af5f44f8a7ca0588d94fbb58242ae788daa27 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.709] Peer's transaction rejected              peer=d6c86efc6657cb027ab912fa53e6188a893991ef6b2ec50c43a93e79e6b0978e txHash=0x173bbdd11ebd1620671aca4bcdabe0a910340214e48d365800e24715cd9c2309 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.749] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0xb165622652a003b2c2ba37c21c1af5f44f8a7ca0588d94fbb58242ae788daa27 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:38.823] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0x0b907de9807fed411f588fb4b2d1dbf96b7b9d6a867c949a9b2db54ade8f1b38 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.035] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0xd638eddbdd789b42f075a2ca2bc96de021a58b176b1e579cecf598c0aee6c349 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.048] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0x634a889d26c5cb1f25a5950e4617f559fed90d0f53609fcd587fd895d734895d err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.202] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0xf76105bd281e467c8d0618269c93a1676d31f26fc365f1788b0a6871aca8aa35 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.613] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0x480d8a94b660d492dacff20fca3031f2b5c324a1869be480cb0662da95d11c0e err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.776] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0x634a889d26c5cb1f25a5950e4617f559fed90d0f53609fcd587fd895d734895d err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.776] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0x859c323e9694321251c517711b71bafef4ac81395aee61767752ace35257cfc9 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:39.779] Peer's transaction rejected              peer=d6c86efc6657cb027ab912fa53e6188a893991ef6b2ec50c43a93e79e6b0978e txHash=0xf8f87710584e08ff113b54d2750abaa8ada27dae76526a07a23278042d67d5ca err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:40.046] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0xd036255b9656e58c03569c45a09aded1c37587f3c48d5bc3877464410684b88d err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:40.317] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0xf8f87710584e08ff113b54d2750abaa8ada27dae76526a07a23278042d67d5ca err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:40.318] Peer's transaction rejected              peer=b644c11efbc20377aa21fa80afa69fb84c556b51d0c4dea49e01331fd22f01ef txHash=0xeb219331efdcc9f9ab9ef8bb08a12d418ea3362405ed20f1ca32d95576b533f3 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:40.928] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0xb64f3aaa5d293d7b40921a49c36afc428e4db27f5456f56f3327d0fb301a49ee err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:40.976] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0xcb9519a5bd5547ac8e709536d9e66bcc2a560fac090d4884cdd137ebaa968651 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:41.095] Peer's transaction rejected              peer=f461626c16eca899a36d140cecde0c624fe618777e26a3a6bdb462cb90c077cb txHash=0xe1d5fe933d6c311c926cf45c83c18db62eed831dad6363dec2fe2b1e8054bf69 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:41.263] Peer's transaction rejected              peer=d6c86efc6657cb027ab912fa53e6188a893991ef6b2ec50c43a93e79e6b0978e txHash=0x5e3ab3d75d596efce0922135e6688ac57033a3a69105071e0faf6a30cc67c3a4 err="insufficient funds for gas * price + value"
11|opbnb-geth  | WARN [03-14|01:02:41.280] Peer's transaction rejected              peer=2f1dcad170b9e4acf3bcb700379fb3d368b9f57d25630aadc5ab3f95be80212f txHash=0x5a4bd10bdcf11384ea88b9649dd7f93c1b821db91c4ca0e5480358acd0b834e9 err="insufficient funds for gas * price + value"

Steps to reproduce the behaviour

start-op-node.sh:

#! /usr/bin/bash

set -ex

cd op-node-data

export L2_RPC=http://localhost:8551

export P2P_PRIV_KEY=0378380d7efc3d813810866286261a3f96b286508c53dc42de4e9d5f368597fb

export L1_RPC=https://bsc-dataseed.bnbchain.org
export P2P_BOOTNODES="enr:-J24QGQBeMsXOaCCaLWtNFSfb2Gv50DjGOKToH2HUTAIn9yXImowlRoMDNuPNhSBZNQGCCE8eAl5O3dsONuuQp5Qix2GAYjB7KHSgmlkgnY0gmlwhDREiqaHb3BzdGFja4PrKwCJc2VjcDI1NmsxoQL4I9wpEVDcUb8bLWu6V8iPoN5w8E8q-GrS5WUCygYUQ4N0Y3CCIyuDdWRwgiMr,enr:-J24QJ>
# --p2p.bootnodes=$P2P_BOOTNODES \

./op-node \
  --l1.trustrpc \
  --sequencer.l1-confs=15 \
  --verifier.l1-confs=15 \
  --l1.http-poll-interval 3s \
  --l1.epoch-poll-interval 45s \
  --l1.rpc-max-batch-size 20 \
  --network=opBNBMainnet \
  --rpc.addr=0.0.0.0 \
  --rpc.port=8546 \
  --p2p.sync.req-resp \
  --p2p.listen.ip=0.0.0.0 \
  --p2p.listen.tcp=9003 \
  --p2p.listen.udp=9003 \
  --snapshotlog.file=./snapshot.log \
  --metrics.enabled \
  --metrics.addr=0.0.0.0 \
  --metrics.port=7300 \
  --pprof.enabled \
  --rpc.enable-admin \
  --l1=${L1_RPC} \
  --l2=${L2_RPC} \
  --l2.jwt-secret=./jwt.txt \
  --l2.engine-sync=true \
  --l2.skip-sync-start-check=true \
  --log.level=debug

start-op-geth.sh:

#! /usr/bin/bash
cd op-geth-data

export CHAIN_ID=204
export L2_RPC=https://opbnb-mainnet-rpc.bnbchain.org
export P2P_BOOTNODES="enr:-KO4QKFOBDW--pF4pFwv3Al_jiLOITj_Y5mr1Ajyy2yxHpFtNcBfkZEkvWUxAKXQjWALZEFxYHooU88JClyzA00e8YeGAYtBOOZig2V0aMfGhE0ZYGqAgmlkgnY0gmlwhDREiqaJc2VjcDI1NmsxoQM8pC_6wwTr5N2Q-yXQ1KGKsgz9i9EPLk8Ata65pUyYG4RzbmFwwIN0Y3CCdl-DdWRwgnZf,en>
#  --bootnodes=$P2P_BOOTNODES

./op-geth \
  --datadir="./datadir" \
  --verbosity=3 \
  --http \
  --http.corsdomain="*" \
  --http.vhosts="*" \
  --http.addr=0.0.0.0 \
  --http.port=8545 \
  --http.api=net,eth,engine \
  --ws \
  --ws.addr=0.0.0.0 \
  --ws.port=8545 \
  --ws.origins="*" \
  --ws.api=eth,engine \
  --syncmode=full \
  --maxpeers=10 \
  --networkid=$CHAIN_ID \
  --miner.gaslimit=150000000 \
  --triesInMemory=32 \
  --txpool.globalslots=10000 \
  --txpool.globalqueue=5000 \
  --txpool.accountqueue=200 \
  --txpool.accountslots=200 \
  --cache 32000 \
  --cache.preimages \
  --allow-insecure-unlock \
  --authrpc.addr="0.0.0.0" \
  --authrpc.port="8551" \
  --authrpc.vhosts="*" \
  --authrpc.jwtsecret=./jwt.txt \
  --gcmode=archive \
  --metrics \
  --metrics.port 6060 \
  --metrics.addr 0.0.0.0 \
  --rollup.sequencerhttp=$L2_RPC
welkin22 commented 6 months ago

@e1ghtball Hi, the log in op-node: pre fetch receipts fail without error, need retry is not important, you can notice that its level is debug, which means that the pre-fetch receipts cache is full, so the pre-fetch work cannot continue and needs to be retried constantly. At the same time, the Peer's transaction rejected in op-geth is also not a big problem, it means that some transactions broadcasted by peers are rejected due to "insufficient funds for gas * price + value" reasons. These logs can also appear in normal nodes. In the logs you provided, only one line is abnormal: lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=153, which means that the derivation.step process of the op-node has failed 153 times. The specific reason for the failure have been printed in an error or warn log in a place that you did not provide. You need to provide more op-node logs so that we can determine where the problem occurred.

e1ghtball commented 6 months ago

@welkin22 is it correct part of logs?

t=2024-03-17T10:37:49+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfe27a1d1b36b277c82e6abbd2b42b18436ac74d60187dbe9f39e90754d3b0974 blockNumber=36,748,957
t=2024-03-17T10:37:49+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfa267d23ed5467de28f2b9faf2cdff4518b18e428c5d8f1f53db2307cef46790 blockNumber=36,748,952
t=2024-03-17T10:37:49+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x518fd71216ac40f04b5f407b3823f835d1abc992ae871173833ab5c4bb484885 blockNumber=36,748,959
t=2024-03-17T10:37:49+0000 lvl=eror msg="Payload execution failed"               block_hash=0x2a43355734071576a72c703f7b795cb2732afbf79edbd04618eb8c4171d76744 err="Post \"http://localhost:8551\": context deadline exceeded"
t=2024-03-17T10:37:49+0000 lvl=warn msg="Derivation process temporary error"     attempts=1 err="engine stage failed: temp: failed to update insert payload: failed to execute payload: Post \"http://localhost:8551\": context deadline exceeded"
t=2024-03-17T10:37:49+0000 lvl=dbug msg="scheduling re-attempt with delay"       attempts=1 delay=2.04s
t=2024-03-17T10:37:49+0000 lvl=dbug msg="L1 head moved forward"                  l1_head=0x8e8135333146af863d3c048aad0321c595eb53c05acbbb813c4c2305a4a26a5f:37045601
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=dbug msg="requesting missing unsafe L2 block range" start=0x296d6788275f2c8d464f1420770b732cf0a811d4faecccfb2d591c2f648c7406:18176880 end=0x2a43355734071576a72c703f7b795cb2732afbf79edbd04618eb8c4171d76744:18918141 size=741,261
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring request to sync L2 range, timestamp is too old for p2p" start=0x296d6788275f2c8d464f1420770b732cf0a811d4faecccfb2d591c2f648c7406:18176880 end=0x2a43355734071576a72c703f7b795cb2732afbf79edbd04618eb8c4171d76744:18918141 start_time=1,709,930,603
t=2024-03-17T10:37:49+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x3606840e4fd878d440c23e4258d0250f04fd23617906ff6844a26e3afc794927:18918142
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xecd1a5b5e3a3493a9e608a6cf206bede030fb70955ff5eef625c6fda3a182c0c:18918143
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xbb4c1fef7d8f429ac9db432685f19b15253245ee329971e70fdb59d02e4f418b:18918144
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xe5291ea7a79e95e542b5c5823bcd61ae25b7c396c566f1a966f6d5d1768feb46:18918145
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xc6a2926aae0ccffa09b55de31e4553d6112ea591efbe3dc65759a1010946450e:18918146
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=dbug msg="L1 head moved forward"                  l1_head=0x62ffce8981a45fa593ef9dfd23d17fd4a69ff7e833ac1df8f3bd03c223f933af:37045602
t=2024-03-17T10:37:49+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:49+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xe696ba2655a3c95fe80e4eb005d633ba73783ddff137079a731db5106fb81725 blockNumber=36,748,961
t=2024-03-17T10:37:49+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xbf534a584a04455ae494693499aec5ffb426d1d15b2ddf0e9f25d288ba380c39 blockNumber=36,748,956
t=2024-03-17T10:37:49+0000 lvl=dbug msg="Revalidated node"                       p2p=discv5 b=15 id=8b59a309783df517e66695d493f75c0ddb15d7c44355c6fec020366a6ece047e checks=2
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x66edcb0ac875de8110164edf77bd7b86871f2c7bf2040e7dde0166fa38f24f00 blockNumber=36,748,960
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x21853c48a1f134f213e8db0e8c5caf63b802eefbd2c1f18ac3ab26984dd9cbdf blockNumber=36,748,958
t=2024-03-17T10:37:50+0000 lvl=dbug msg="peering tick"                           connected=28 advertised_udp=9003 advertised_tcp=9003 advertised_ip=23.153.72.203
t=2024-03-17T10:37:50+0000 lvl=dbug msg="validated block"                        result=ACCEPT from=16Uiu2HAm842xSEbJhpL1ZSebb3yft9C3zLnrQpb3d4idgi1bLNzY
t=2024-03-17T10:37:50+0000 lvl=info msg="Received signed execution payload from p2p" id=0xc9c653059db74d7a95e6273bb05629b78e1e2bae8994ab622e81899cae3a50c4:18918147 peer=16Uiu2HAm842xSEbJhpL1ZSebb3yft9C3zLnrQpb3d4idgi1bLNzY
t=2024-03-17T10:37:50+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xc9c653059db74d7a95e6273bb05629b78e1e2bae8994ab622e81899cae3a50c4:18918147
t=2024-03-17T10:37:50+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xdfbdcdc53ca64adae5b490639b087e61915bdca7b88e334fab6e016ced3bc5a7 blockNumber=36,748,953
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x79ded6f894377dd27f3ccedd61c9fd13ed3ecc33b8ed878de95e9bda57770b33 blockNumber=36,748,955
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x45dc47a8d3a03f60813fcf213e4199740df5775d9e9cf4e8082c42819369513f blockNumber=36,748,954
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfe27a1d1b36b277c82e6abbd2b42b18436ac74d60187dbe9f39e90754d3b0974 blockNumber=36,748,957
t=2024-03-17T10:37:50+0000 lvl=dbug msg="requesting missing unsafe L2 block range" start=0x296d6788275f2c8d464f1420770b732cf0a811d4faecccfb2d591c2f648c7406:18176880 end=0x2a43355734071576a72c703f7b795cb2732afbf79edbd04618eb8c4171d76744:18918141 size=741,261
t=2024-03-17T10:37:50+0000 lvl=dbug msg="ignoring request to sync L2 range, timestamp is too old for p2p" start=0x296d6788275f2c8d464f1420770b732cf0a811d4faecccfb2d591c2f648c7406:18176880 end=0x2a43355734071576a72c703f7b795cb2732afbf79edbd04618eb8c4171d76744:18918141 start_time=1,709,930,603
t=2024-03-17T10:37:50+0000 lvl=info msg="New L1 finalized block"                 l1_finalized=0x65c84a4323ca95c58ffa2c4fbe5c3ca5149dca4d4d9e9eadec3e7c7aa217cabf:37045600
t=2024-03-17T10:37:50+0000 lvl=info msg="received L1 finality signal, but missing data for immediate L2 finalization" prev_finalized_l1=0x65c84a4323ca95c58ffa2c4fbe5c3ca5149dca4d4d9e9eadec3e7c7aa217cabf:37045600 signaled_finalized_l1=0x65c84a4323ca95c58ffa2c4fbe5c3ca5149dca4d4d9e9eadec3e7c7aa217cabf:37045600
t=2024-03-17T10:37:50+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:50+0000 lvl=info msg="New L1 safe block"                      l1_safe=0x8e8135333146af863d3c048aad0321c595eb53c05acbbb813c4c2305a4a26a5f:37045601
t=2024-03-17T10:37:50+0000 lvl=dbug msg="discovered peer"                        peer=16Uiu2HAmSDQYV7jnXRVNPgbM7VeCfGWBCfNngpbEmQ3rDTtzM6o4 nodeID=7044568b6b1df0444ccb989570a27ada097adab67198ce1f74f5c0e387fd5cb3 addr=/ip4/52.193.218.151/tcp/9003
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfa267d23ed5467de28f2b9faf2cdff4518b18e428c5d8f1f53db2307cef46790 blockNumber=36,748,952
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x518fd71216ac40f04b5f407b3823f835d1abc992ae871173833ab5c4bb484885 blockNumber=36,748,959
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xe696ba2655a3c95fe80e4eb005d633ba73783ddff137079a731db5106fb81725 blockNumber=36,748,961
t=2024-03-17T10:37:50+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xbf534a584a04455ae494693499aec5ffb426d1d15b2ddf0e9f25d288ba380c39 blockNumber=36,748,956
t=2024-03-17T10:37:51+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x66edcb0ac875de8110164edf77bd7b86871f2c7bf2040e7dde0166fa38f24f00 blockNumber=36,748,960
t=2024-03-17T10:37:51+0000 lvl=dbug msg="validated block"                        result=ACCEPT from=16Uiu2HAmBQuEvcdUjzFPoY7mKiBUKthwVHjWTnmiVtgQ2MqRnrrH
t=2024-03-17T10:37:51+0000 lvl=info msg="Received signed execution payload from p2p" id=0xd0bd52d77c3da5827346738fdbb1069327ee1f48359e09cdf9987a66447fa804:18918148 peer=16Uiu2HAmBQuEvcdUjzFPoY7mKiBUKthwVHjWTnmiVtgQ2MqRnrrH
t=2024-03-17T10:37:51+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xd0bd52d77c3da5827346738fdbb1069327ee1f48359e09cdf9987a66447fa804:18918148
t=2024-03-17T10:37:51+0000 lvl=dbug msg="ignoring step request, already scheduled re-attempt after previous failure" attempts=1
t=2024-03-17T10:37:51+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x21853c48a1f134f213e8db0e8c5caf63b802eefbd2c1f18ac3ab26984dd9cbdf blockNumber=36,748,958
t=2024-03-17T10:37:51+0000 lvl=dbug msg="Removed dead node"                      p2p=discv5 b=14 id=cb59dbfad4d76e3dc48c67c822a92e5006df7c17234be1a7cb6cd172d9e8f43a ip=195.201.71.223  checks=0
t=2024-03-17T10:37:51+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xdfbdcdc53ca64adae5b490639b087e61915bdca7b88e334fab6e016ced3bc5a7 blockNumber=36,748,953
t=2024-03-17T10:37:51+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x79ded6f894377dd27f3ccedd61c9fd13ed3ecc33b8ed878de95e9bda57770b33 blockNumber=36,748,955
t=2024-03-17T10:37:51+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0x45dc47a8d3a03f60813fcf213e4199740df5775d9e9cf4e8082c42819369513f blockNumber=36,748,954
t=2024-03-17T10:37:51+0000 lvl=dbug msg="pre fetch receipts fail without error,need retry" blockHash=0xfe27a1d1b36b277c82e6abbd2b42b18436ac74d60187dbe9f39e90754d3b0974 blockNumber=36,748,957
welkin22 commented 6 months ago

@e1ghtball Yes, the useful one is the following:

t=2024-03-17T10:37:49+0000 lvl=eror msg="Payload execution failed"               block_hash=0x2a43355734071576a72c703f7b795cb2732afbf79edbd04618eb8c4171d76744 err="Post \"http://localhost:8551\": context deadline exceeded"

This means that your op-geth seems to have some problems, you can check the op-geth log to confirm the issue. Based on recent cases I have seen, I guess you may have encountered a problem related to db compaction. You can search for the keyword "Database compacting, degraded performance" in the op-geth logs. If you can find it, please follow the instructions in this document to try to fix the issue.

e1ghtball commented 6 months ago

@welkin22 yes, I see logs like Database compacting, degraded performance every minute. And yes, I'm running archive node as described in document that you shared. But specs of my machine are pretty good: Ryzen 9 5900X (12 cores/24 threads), 128GB RAM. Also, I don't see any CPU load at all - screenshot from btop Any chances to fix this problem without switching to pruned node?

welkin22 commented 6 months ago

@welkin22 yes, I see logs like Database compacting, degraded performance every minute. And yes, I'm running archive node as described in document that you shared. But specs of my machine are pretty good: Ryzen 9 5900X (12 cores/24 threads), 128GB RAM. Also, I don't see any CPU load at all - screenshot from btop Any chances to fix this problem without switching to pruned node?

@e1ghtball The key indicator is the performance of the disk, and you need to pay attention to the disk's throughput capacity. Your CPU and memory are already sufficient.

e1ghtball commented 6 months ago

@welkin22 I tested geth working on Hetzner machines, which should definitely not have weak SSDs - the problem persisted When I upgraded to version 0.3.0, the problem persisted too

The problem only resolved with upgrading to 0.3.1 alpha and using pebble db. There were no snapshots for it, I had to wait until the state synchronises from scratch, so I write with such a delay

welkin22 commented 6 months ago

@welkin22 I tested geth working on Hetzner machines, which should definitely not have weak SSDs - the problem persisted When I upgraded to version 0.3.0, the problem persisted too

The problem only resolved with upgrading to 0.3.1 alpha and using pebble db. There were no snapshots for it, I had to wait until the state synchronises from scratch, so I write with such a delay

@e1ghtball Thank you for your feedback. 0.3.1 alpha is still in the public testing phase, and when we finally release it, we will plan to provide snapshots to help users quickly start new nodes.

mateipopp commented 5 months ago

Hello,

I'm experiencing the same issue on our end with database compacting and degraded performance. We are using NVMe disks with a RAID 0 configuration, Intel Xeon-E 2388G - 8c/16t - 3.2GHz/4.6GHz, and 64GB DDR4 ECC 3200MHz. When do you think the Pebble DB snapshot will be released?

Thanks

welkin22 commented 3 months ago

Hello,

I'm experiencing the same issue on our end with database compacting and degraded performance. We are using NVMe disks with a RAID 0 configuration, Intel Xeon-E 2388G - 8c/16t - 3.2GHz/4.6GHz, and 64GB DDR4 ECC 3200MHz. When do you think the Pebble DB snapshot will be released?

Thanks

@mateipopp Hello, if you want to run an archive node with Pebble DB, we recommend syncing a new node from scratch. Currently, we do not have plans to provide an archive snapshot.