search

paradigmxyz / reth

Modular, contributor-friendly and blazing-fast implementation of the Ethereum protocol, in Rust
https://reth.rs/
Apache License 2.0
3.9k stars 1.14k forks source link

Extended elapsed time #7519

Closed mrabino1 closed 2 months ago

mrabino1 commented 6 months ago

Describe the bug

see screen snapshot

2024-04-09_10-38-30

Steps to reproduce

run beta.5

Node logs

see log

Platform(s)

Linux (x86)

What version/commit are you on?

beta.5

What database version are you on?

beta.5

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

shekhirin commented 6 months ago

Can you please show the logs from ~/.cache/reth/logs/mainnet/reth.log? They should shed the light on what's taking so long.

mrabino1 commented 6 months ago 
2024-04-09T13:59:01.361106Z DEBUG blockchain_tree: Committing new canonical chain: [(19618534, 0x81b40181caacde9718c1a04844f15e09c15542261e99360646dad789e7993b91)]
2024-04-09T13:59:01.362641Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618534, 0x81b40181caacde9718c1a04844f15e09c15542261e99360646dad789e7993b91)]
2024-04-09T13:59:01.398947Z DEBUG providers::db: Inserted block block_number=19618534 actions=[(InsertCanonicalHeaders, 57.223µs), (InsertHeaders, 27.304µs), (InsertHeaderNumbers, 62.668µs), (GetParentTD, 1.165µs), (InsertHeaderTerminalDifficulties, 14.907µs), (GetNextTxNum, 5.893µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 824.134µs), (InsertTransactionHashNumbers, 35.146442ms), (InsertBlockWithdrawals, 39.372µs), (InsertBlockBodyIndices, 28.965µs), (InsertTransactionBlocks, 31.09µs)]
2024-04-09T13:59:01.514674Z DEBUG providers::db: Appended blocks range=19618534..=19618534 actions=[(InsertBlock, 36.290391ms), (InsertState, 13.630383ms), (InsertHashes, 59.893247ms), (InsertHistoryIndices, 42.161858ms), (UpdatePipelineStages, 22.085µs)]
2024-04-09T13:59:01.554464Z DEBUG storage::db::mdbx: Commit total_duration=39.774118ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 16, audit: 0, write: 0, sync: 2588, ending: 0, whole: 2605, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T13:59:01.554491Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 19.015µs), (FindCanonicalHeader, 14.092µs), (SplitChain, 1.701µs), (SplitChainForks, 1.072µs), (MergeAllChains, 1.138µs), (UpdateCanonicalIndex, 3.953µs), (RetrieveStateTrieUpdates, 1.545516ms), (CommitCanonicalChainToDatabase, 191.834429ms)]
2024-04-09T13:59:01.554531Z DEBUG consensus::engine: Canonicalized new head hash=0x81b40181caacde9718c1a04844f15e09c15542261e99360646dad789e7993b91 number=19618534
2024-04-09T13:59:01.554583Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:01.554592Z  INFO reth_node_core::events::node: Canonical chain committed number=19618534 hash=0x81b40181caacde9718c1a04844f15e09c15542261e99360646dad789e7993b91 elapsed=193.458237ms
2024-04-09T13:59:13.035797Z DEBUG blockchain_tree: Appending block to canonical chain head=0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013 parent=(19618534, 0x81b40181caacde9718c1a04844f15e09c15542261e99360646dad789e7993b91)
2024-04-09T13:59:13.165199Z DEBUG evm: Execution time evm_transact=127.236009ms apply_state=446.968µs apply_post_state=7.495µs merge_transitions=358.046µs receipt_root=896.069µs
2024-04-09T13:59:13.166230Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=328
2024-04-09T13:59:14.137158Z DEBUG blockchain_tree::chain: Validated state root number=19618535 hash=0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013 elapsed=971.945505ms
2024-04-09T13:59:14.137261Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-09T13:59:14.137294Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618535 hash=0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013 peers=100 txs=183 mgas=11.655 full=38.8% base_fee=37.95gwei blobs=0 excess_blobs=0 elapsed=1.114892433s
2024-04-09T13:59:14.337915Z DEBUG blockchain_tree: Committing new canonical chain: [(19618535, 0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013)]
2024-04-09T13:59:14.340557Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618535, 0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013)]
2024-04-09T13:59:14.402525Z DEBUG providers::db: Inserted block block_number=19618535 actions=[(InsertCanonicalHeaders, 78.744µs), (InsertHeaders, 41.808µs), (InsertHeaderNumbers, 84.212µs), (GetParentTD, 1.918µs), (InsertHeaderTerminalDifficulties, 19.519µs), (GetNextTxNum, 6.821µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 1.324177ms), (InsertTransactionHashNumbers, 59.631817ms), (InsertBlockWithdrawals, 36.975µs), (InsertBlockBodyIndices, 31.937µs), (InsertTransactionBlocks, 31.728µs)]
2024-04-09T13:59:14.577693Z DEBUG providers::db: Appended blocks range=19618535..=19618535 actions=[(InsertBlock, 61.352892ms), (InsertState, 18.799691ms), (InsertHashes, 92.366928ms), (InsertHistoryIndices, 63.930597ms), (UpdatePipelineStages, 51.451µs)]
2024-04-09T13:59:14.631603Z DEBUG storage::db::mdbx: Commit total_duration=53.893138ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 14, audit: 0, write: 0, sync: 3512, ending: 0, whole: 3527, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T13:59:14.631635Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 23.255µs), (FindCanonicalHeader, 14.401µs), (SplitChain, 2.497µs), (SplitChainForks, 1.42µs), (MergeAllChains, 1.3µs), (UpdateCanonicalIndex, 5.115µs), (RetrieveStateTrieUpdates, 2.70051ms), (CommitCanonicalChainToDatabase, 291.013707ms)]
2024-04-09T13:59:14.631672Z DEBUG consensus::engine: Canonicalized new head hash=0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013 number=19618535
2024-04-09T13:59:14.631725Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:14.631752Z  INFO reth_node_core::events::node: Canonical chain committed number=19618535 hash=0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013 elapsed=293.806594ms
2024-04-09T13:59:25.194775Z DEBUG blockchain_tree: Appending block to canonical chain head=0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3 parent=(19618535, 0x754bf05324e3ff94d6f7ac866657d2e1941ecc5dc26683ac770e9b768ea92013)
2024-04-09T13:59:25.345552Z DEBUG evm: Execution time evm_transact=148.992472ms apply_state=372.721µs apply_post_state=8.192µs merge_transitions=249.89µs receipt_root=810.681µs
2024-04-09T13:59:25.346862Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=260
2024-04-09T13:59:26.098861Z DEBUG blockchain_tree::chain: Validated state root number=19618536 hash=0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3 elapsed=753.294478ms
2024-04-09T13:59:26.098969Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-09T13:59:26.098993Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618536 hash=0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3 peers=100 txs=144 mgas=10.422 full=34.7% base_fee=36.89gwei blobs=6 excess_blobs=0 elapsed=917.030315ms
2024-04-09T13:59:26.282929Z DEBUG blockchain_tree: Committing new canonical chain: [(19618536, 0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3)]
2024-04-09T13:59:26.284750Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618536, 0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3)]
2024-04-09T13:59:26.335323Z DEBUG providers::db: Inserted block block_number=19618536 actions=[(InsertCanonicalHeaders, 62.452µs), (InsertHeaders, 19.982µs), (InsertHeaderNumbers, 45.506µs), (GetParentTD, 1.107µs), (InsertHeaderTerminalDifficulties, 13.49µs), (GetNextTxNum, 5.548µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 1.221048ms), (InsertTransactionHashNumbers, 48.949608ms), (InsertBlockWithdrawals, 35.887µs), (InsertBlockBodyIndices, 32.689µs), (InsertTransactionBlocks, 30.013µs)]
2024-04-09T13:59:26.492712Z DEBUG providers::db: Appended blocks range=19618536..=19618536 actions=[(InsertBlock, 50.478881ms), (InsertState, 17.662776ms), (InsertHashes, 87.968463ms), (InsertHistoryIndices, 51.712439ms), (UpdatePipelineStages, 25.65µs)]
2024-04-09T13:59:26.534051Z DEBUG storage::db::mdbx: Commit total_duration=41.322465ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 13, audit: 0, write: 0, sync: 2687, ending: 0, whole: 2701, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T13:59:26.534081Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 20.375µs), (FindCanonicalHeader, 12.836µs), (SplitChain, 2.141µs), (SplitChainForks, 1.057µs), (MergeAllChains, 1.067µs), (UpdateCanonicalIndex, 4.015µs), (RetrieveStateTrieUpdates, 1.829923ms), (CommitCanonicalChainToDatabase, 249.314354ms)]
2024-04-09T13:59:26.534111Z DEBUG consensus::engine: Canonicalized new head hash=0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3 number=19618536
2024-04-09T13:59:26.534162Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:26.534169Z  INFO reth_node_core::events::node: Canonical chain committed number=19618536 hash=0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3 elapsed=251.225412ms
2024-04-09T13:59:36.998674Z DEBUG blockchain_tree: Appending block to canonical chain head=0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519 parent=(19618536, 0x357e6b6810dee0571c2151d285bf35fa5ca36b95961c6f1ac4ce0f2699caa8f3)
2024-04-09T13:59:37.242840Z DEBUG evm: Execution time evm_transact=239.748565ms apply_state=1.05118ms apply_post_state=7.544µs merge_transitions=746.7µs receipt_root=2.17275ms
2024-04-09T13:59:37.244841Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=553
2024-04-09T13:59:38.946742Z DEBUG blockchain_tree::chain: Validated state root number=19618537 hash=0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519 elapsed=1.703888417s
2024-04-09T13:59:38.946903Z DEBUG pruner: Minimum pruning interval reached previous_tip_block_number=Some(19618531) tip_block_number=19618536
2024-04-09T13:59:38.946938Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618537 hash=0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519 peers=100 txs=317 mgas=29.987 full=100.0% base_fee=35.48gwei blobs=3 excess_blobs=3 elapsed=1.953025466s
2024-04-09T13:59:38.946945Z DEBUG consensus::engine::hooks: Polled next hook hook="Prune" result=PolledHook { name: "Prune", event: Started, db_access_level: ReadWrite }
2024-04-09T13:59:38.946956Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:38.947013Z DEBUG pruner: Pruner started tip_block_number=19618536
2024-04-09T13:59:38.947056Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=19618454 prune_mode=Before(19618455)
2024-04-09T13:59:38.947093Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=19618454 prune_mode=Before(19618455) output.pruned=0
2024-04-09T13:59:38.947101Z  INFO reth_node_core::events::node: Pruner started tip_block_number=19618536
2024-04-09T13:59:38.947102Z DEBUG pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=19618454 prune_mode=Before(19618455)
2024-04-09T13:59:38.947106Z DEBUG pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=19618454 prune_mode=Before(19618455) output.pruned=0
2024-04-09T13:59:38.947108Z DEBUG pruner: Segment pruning started segment=Receipts purpose=User to_block=11052983 prune_mode=Before(11052984)
2024-04-09T13:59:38.947112Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=User to_block=11052983 prune_mode=Before(11052984) output.pruned=0
2024-04-09T13:59:38.947115Z DEBUG pruner: Nothing to prune for the segment segment=ContractLogs purpose=User
2024-04-09T13:59:38.947117Z DEBUG pruner: Segment pruning started segment=SenderRecovery purpose=User to_block=19618536 prune_mode=Full
2024-04-09T13:59:38.954822Z DEBUG providers::db: Pruning limit reached limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 17500, deleted: 17500 }), time_limit: Some(PruneTimeLimit { limit: 100ms, start: Instant { tv_sec: 341640, tv_nsec: 730750418 } }) } deleted_entries_limit=true time_limit=false table=TransactionSenders
2024-04-09T13:59:38.954967Z DEBUG pruner: Segment pruning finished segment=SenderRecovery purpose=User to_block=19618536 prune_mode=Full output.pruned=17500
2024-04-09T13:59:38.956572Z DEBUG storage::db::mdbx: Commit total_duration=1.597524ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 2, audit: 0, write: 0, sync: 101, ending: 0, whole: 103, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T13:59:38.956588Z DEBUG pruner: Pruner interrupted and has more data to prune tip_block_number=19618536 elapsed=9.550583ms deleted_entries=17500 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 17500, deleted: 17500 }), time_limit: Some(PruneTimeLimit { limit: 100ms, start: Instant { tv_sec: 341640, tv_nsec: 730750418 } }) } progress=HasMoreData(DeletedEntriesLimitReached) stats={SenderRecovery: (HasMoreData(DeletedEntriesLimitReached), 17500)}
2024-04-09T13:59:38.956607Z DEBUG consensus::engine::hooks: Polled running hook with db write access hook="Prune" result=PolledHook { name: "Prune", event: Finished(Ok(())), db_access_level: ReadWrite }
2024-04-09T13:59:38.956619Z  INFO reth_node_core::events::node: Pruner finished tip_block_number=19618536 elapsed=9.550583ms stats={SenderRecovery: (HasMoreData(DeletedEntriesLimitReached), 17500)}
2024-04-09T13:59:38.956775Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:39.157641Z DEBUG blockchain_tree: Committing new canonical chain: [(19618537, 0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519)]
2024-04-09T13:59:39.161423Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618537, 0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519)]
2024-04-09T13:59:39.288493Z DEBUG providers::db: Inserted block block_number=19618537 actions=[(InsertCanonicalHeaders, 41.629µs), (InsertHeaders, 31.703µs), (InsertHeaderNumbers, 34.117µs), (GetParentTD, 913ns), (InsertHeaderTerminalDifficulties, 10.92µs), (GetNextTxNum, 5.45µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 18.673253ms), (InsertTransactionHashNumbers, 108.080696ms), (InsertBlockWithdrawals, 28.183µs), (InsertBlockBodyIndices, 17.978µs), (InsertTransactionBlocks, 23.489µs)]
2024-04-09T13:59:39.621198Z DEBUG providers::db: Appended blocks range=19618537..=19618537 actions=[(InsertBlock, 127.051681ms), (InsertState, 33.47367ms), (InsertHashes, 179.331225ms), (InsertHistoryIndices, 119.852325ms), (UpdatePipelineStages, 27.184µs)]
2024-04-09T13:59:39.708942Z DEBUG storage::db::mdbx: Commit total_duration=87.72686ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 102, audit: 0, write: 0, sync: 5639, ending: 5, whole: 5748, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T13:59:39.708968Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 21.903µs), (FindCanonicalHeader, 11.233µs), (SplitChain, 2.345µs), (SplitChainForks, 1.332µs), (MergeAllChains, 1.447µs), (UpdateCanonicalIndex, 5.314µs), (RetrieveStateTrieUpdates, 3.796423ms), (CommitCanonicalChainToDatabase, 547.528066ms)]
2024-04-09T13:59:39.708994Z DEBUG consensus::engine: Canonicalized new head hash=0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519 number=19618537
2024-04-09T13:59:39.709026Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-09T13:59:39.709122Z  INFO reth_node_core::events::node: Canonical chain committed number=19618537 hash=0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519 elapsed=551.400925ms
2024-04-09T13:59:49.044150Z DEBUG blockchain_tree: Appending block to canonical chain head=0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b parent=(19618537, 0x15e4bdc943b30bb20d98c5b4557fc92d69adcfca753f766208f4840b383a1519)
2024-04-09T13:59:49.167894Z DEBUG evm: Execution time evm_transact=121.858587ms apply_state=442.2µs apply_post_state=7.105µs merge_transitions=340.024µs receipt_root=960.217µs
2024-04-09T13:59:49.168885Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=253
2024-04-09T13:59:50.130848Z DEBUG blockchain_tree::chain: Validated state root number=19618538 hash=0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b elapsed=962.941013ms
2024-04-09T13:59:50.130985Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:50.131013Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618538 hash=0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b peers=100 txs=117 mgas=11.762 full=39.2% base_fee=39.91gwei blobs=0 excess_blobs=3 elapsed=1.089502567s
2024-04-09T13:59:50.318276Z DEBUG blockchain_tree: Committing new canonical chain: [(19618538, 0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b)]
2024-04-09T13:59:50.320096Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618538, 0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b)]
2024-04-09T13:59:50.360936Z DEBUG providers::db: Inserted block block_number=19618538 actions=[(InsertCanonicalHeaders, 50.275µs), (InsertHeaders, 18.325µs), (InsertHeaderNumbers, 31.709µs), (GetParentTD, 955ns), (InsertHeaderTerminalDifficulties, 9.407µs), (GetNextTxNum, 5.387µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 1.182774ms), (InsertTransactionHashNumbers, 39.373673ms), (InsertBlockWithdrawals, 37.47µs), (InsertBlockBodyIndices, 31.423µs), (InsertTransactionBlocks, 29.902µs)]
2024-04-09T13:59:50.518984Z DEBUG providers::db: Appended blocks range=19618538..=19618538 actions=[(InsertBlock, 40.825524ms), (InsertState, 15.952588ms), (InsertHashes, 85.765939ms), (InsertHistoryIndices, 56.286369ms), (UpdatePipelineStages, 23.516µs)]
2024-04-09T13:59:50.562434Z DEBUG storage::db::mdbx: Commit total_duration=43.432776ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 15, audit: 0, write: 0, sync: 2824, ending: 1, whole: 2841, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T13:59:50.562461Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 20.582µs), (FindCanonicalHeader, 12.04µs), (SplitChain, 1.582µs), (SplitChainForks, 1.067µs), (MergeAllChains, 1.114µs), (UpdateCanonicalIndex, 3.391µs), (RetrieveStateTrieUpdates, 1.831455ms), (CommitCanonicalChainToDatabase, 242.349794ms)]
2024-04-09T13:59:50.562486Z DEBUG consensus::engine: Canonicalized new head hash=0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b number=19618538
2024-04-09T13:59:50.562518Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-09T13:59:50.562642Z  INFO reth_node_core::events::node: Canonical chain committed number=19618538 hash=0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b elapsed=244.25391ms
2024-04-09T13:59:59.577248Z DEBUG blockchain_tree: Appending block to canonical chain head=0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3 parent=(19618538, 0x88535a2db5f891f89833a8636d5fd9c5302fdb7c254652d91db7c157c961125b)
2024-04-09T13:59:59.623518Z DEBUG evm: Execution time evm_transact=45.31121ms apply_state=143.677µs apply_post_state=9.856µs merge_transitions=85.969µs receipt_root=294.703µs
2024-04-09T13:59:59.623881Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=109
2024-04-09T13:59:59.979390Z DEBUG blockchain_tree::chain: Validated state root number=19618539 hash=0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3 elapsed=355.858376ms
2024-04-09T13:59:59.979464Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T13:59:59.979500Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618539 hash=0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3 peers=100 txs=56 mgas=3.584 full=11.9% base_fee=38.84gwei blobs=6 excess_blobs=0 elapsed=414.250887ms
2024-04-09T14:00:00.146365Z DEBUG blockchain_tree: Committing new canonical chain: [(19618539, 0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3)]
2024-04-09T14:00:00.146938Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618539, 0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3)]
2024-04-09T14:00:00.165954Z DEBUG providers::db: Inserted block block_number=19618539 actions=[(InsertCanonicalHeaders, 48.57µs), (InsertHeaders, 20.346µs), (InsertHeaderNumbers, 31.018µs), (GetParentTD, 1.104µs), (InsertHeaderTerminalDifficulties, 10.47µs), (GetNextTxNum, 5.169µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 378.406µs), (InsertTransactionHashNumbers, 18.33865ms), (InsertBlockWithdrawals, 34.302µs), (InsertBlockBodyIndices, 30.493µs), (InsertTransactionBlocks, 27.297µs)]
2024-04-09T14:00:00.222873Z DEBUG providers::db: Appended blocks range=19618539..=19618539 actions=[(InsertBlock, 18.960048ms), (InsertState, 5.340879ms), (InsertHashes, 35.860871ms), (InsertHistoryIndices, 15.675023ms), (UpdatePipelineStages, 24.447µs)]
2024-04-09T14:00:00.244724Z DEBUG storage::db::mdbx: Commit total_duration=21.833381ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 10, audit: 0, write: 0, sync: 1416, ending: 0, whole: 1427, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T14:00:00.244751Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 17.235µs), (FindCanonicalHeader, 9.567µs), (SplitChain, 1.514µs), (SplitChainForks, 1.06µs), (MergeAllChains, 1.319µs), (UpdateCanonicalIndex, 3.073µs), (RetrieveStateTrieUpdates, 582.147µs), (CommitCanonicalChainToDatabase, 97.799468ms)]
2024-04-09T14:00:00.244785Z DEBUG consensus::engine: Canonicalized new head hash=0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3 number=19618539
2024-04-09T14:00:00.244860Z  INFO reth_node_core::events::node: Canonical chain committed number=19618539 hash=0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3 elapsed=98.456395ms
2024-04-09T14:00:00.244925Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-09T14:00:13.229139Z DEBUG blockchain_tree: Appending block to canonical chain head=0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467 parent=(19618539, 0x89324b364d5ebce2c17e4a95f167b00bcbdc54f81a4b00f26bb5d27f65d38ab3)
2024-04-09T14:00:13.606938Z DEBUG evm: Execution time evm_transact=370.838271ms apply_state=2.017438ms apply_post_state=8.684µs merge_transitions=1.008716ms receipt_root=2.871373ms
2024-04-09T14:00:13.609281Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=712
2024-04-09T14:00:15.930108Z DEBUG blockchain_tree::chain: Validated state root number=19618540 hash=0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467 elapsed=2.323157314s
2024-04-09T14:00:15.930447Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T14:00:15.930498Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618540 hash=0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467 peers=100 txs=419 mgas=29.997 full=100.0% base_fee=35.14gwei blobs=0 excess_blobs=3 elapsed=2.726967122s
2024-04-09T14:00:16.139957Z DEBUG blockchain_tree: Committing new canonical chain: [(19618540, 0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467)]
2024-04-09T14:00:16.147906Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618540, 0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467)]
2024-04-09T14:00:16.294043Z DEBUG providers::db: Inserted block block_number=19618540 actions=[(InsertCanonicalHeaders, 70.128µs), (InsertHeaders, 26.164µs), (InsertHeaderNumbers, 67.801µs), (GetParentTD, 1.528µs), (InsertHeaderTerminalDifficulties, 16.561µs), (GetNextTxNum, 7.473µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 4.059655ms), (InsertTransactionHashNumbers, 141.629712ms), (InsertBlockWithdrawals, 44.47µs), (InsertBlockBodyIndices, 33.495µs), (InsertTransactionBlocks, 33.436µs)]
2024-04-09T14:00:16.701797Z DEBUG providers::db: Appended blocks range=19618540..=19618540 actions=[(InsertBlock, 146.11326ms), (InsertState, 49.044764ms), (InsertHashes, 222.827914ms), (InsertHistoryIndices, 135.840745ms), (UpdatePipelineStages, 21.966µs)]
2024-04-09T14:00:16.827292Z DEBUG storage::db::mdbx: Commit total_duration=125.478801ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 31, audit: 0, write: 0, sync: 8183, ending: 3, whole: 8218, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T14:00:16.827318Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 20.579µs), (FindCanonicalHeader, 14.947µs), (SplitChain, 2.17µs), (SplitChainForks, 1.276µs), (MergeAllChains, 1.242µs), (UpdateCanonicalIndex, 4.281µs), (RetrieveStateTrieUpdates, 7.965164ms), (CommitCanonicalChainToDatabase, 679.391499ms)]
2024-04-09T14:00:16.827342Z DEBUG consensus::engine: Canonicalized new head hash=0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467 number=19618540
2024-04-09T14:00:16.827372Z DEBUG consensus::engine::hooks: Next hook is not ready hook="Prune"
2024-04-09T14:00:16.827448Z  INFO reth_node_core::events::node: Canonical chain committed number=19618540 hash=0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467 elapsed=687.433525ms
2024-04-09T14:00:25.232244Z DEBUG blockchain_tree: Appending block to canonical chain head=0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5 parent=(19618540, 0x52886c1e3c5fbe7e116b040637d30f2bc50ba5bb6c718dc82659492d14e19467)
2024-04-09T14:00:25.594997Z DEBUG evm: Execution time evm_transact=358.018947ms apply_state=1.096105ms apply_post_state=9.222µs merge_transitions=1.208103ms receipt_root=2.054191ms
2024-04-09T14:00:25.597696Z DEBUG trie::parallel_state_root: pre-calculating storage roots len=521
2024-04-09T14:00:27.445808Z DEBUG blockchain_tree::chain: Validated state root number=19618541 hash=0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5 elapsed=1.85079654s
2024-04-09T14:00:27.446014Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T14:00:27.446041Z  INFO reth_node_core::events::node: Block added to canonical chain number=19618541 hash=0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5 peers=100 txs=271 mgas=29.970 full=99.9% base_fee=39.53gwei blobs=6 excess_blobs=0 elapsed=2.219097101s
2024-04-09T14:00:27.664466Z DEBUG blockchain_tree: Committing new canonical chain: [(19618541, 0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5)]
2024-04-09T14:00:27.670515Z DEBUG blockchain_tree: Using cached trie updates blocks=[(19618541, 0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5)]
2024-04-09T14:00:27.770925Z DEBUG providers::db: Inserted block block_number=19618541 actions=[(InsertCanonicalHeaders, 65.004µs), (InsertHeaders, 33.788µs), (InsertHeaderNumbers, 58.39µs), (GetParentTD, 1.266µs), (InsertHeaderTerminalDifficulties, 15.141µs), (GetNextTxNum, 6.081µs), (InsertTransactionSenders, 0ns), (InsertTransactions, 5.08191ms), (InsertTransactionHashNumbers, 94.950634ms), (InsertBlockWithdrawals, 32.171µs), (InsertBlockBodyIndices, 27.912µs), (InsertTransactionBlocks, 29.147µs)]
2024-04-09T14:00:28.067991Z DEBUG providers::db: Appended blocks range=19618541..=19618541 actions=[(InsertBlock, 100.390897ms), (InsertState, 34.076689ms), (InsertHashes, 146.900538ms), (InsertHistoryIndices, 116.049161ms), (UpdatePipelineStages, 20.301µs)]
2024-04-09T14:00:28.136204Z DEBUG storage::db::mdbx: Commit total_duration=68.196367ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 55, audit: 0, write: 0, sync: 4402, ending: 6, whole: 4464, gc_cputime: 15, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T14:00:28.136231Z DEBUG blockchain_tree: Canonicalization finished actions=[(CloneOldBlocks, 27.921µs), (FindCanonicalHeader, 12.198µs), (SplitChain, 2.417µs), (SplitChainForks, 1.926µs), (MergeAllChains, 1.705µs), (UpdateCanonicalIndex, 5.139µs), (RetrieveStateTrieUpdates, 6.066164ms), (CommitCanonicalChainToDatabase, 465.69607ms)]
2024-04-09T14:00:28.136264Z DEBUG consensus::engine: Canonicalized new head hash=0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5 number=19618541
2024-04-09T14:00:28.136312Z DEBUG pruner: Minimum pruning interval reached previous_tip_block_number=Some(19618536) tip_block_number=19618541
2024-04-09T14:00:28.136361Z  INFO reth_node_core::events::node: Canonical chain committed number=19618541 hash=0xb2c2555dfacc57bb062eb30b10b4e1ddcbb3c8cc4af47c642de7a99179a1e9d5 elapsed=471.854056ms
2024-04-09T14:00:28.136376Z DEBUG consensus::engine::hooks: Polled next hook hook="Prune" result=PolledHook { name: "Prune", event: Started, db_access_level: ReadWrite }
2024-04-09T14:00:28.136388Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
2024-04-09T14:00:28.136506Z  INFO reth_node_core::events::node: Pruner started tip_block_number=19618541
2024-04-09T14:00:28.136488Z DEBUG pruner: Pruner started tip_block_number=19618541
2024-04-09T14:00:28.136547Z DEBUG pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=19618454 prune_mode=Before(19618455)
2024-04-09T14:00:28.136627Z DEBUG pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=19618454 prune_mode=Before(19618455) output.pruned=0
2024-04-09T14:00:28.136637Z DEBUG pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=19618454 prune_mode=Before(19618455)
2024-04-09T14:00:28.136643Z DEBUG pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=19618454 prune_mode=Before(19618455) output.pruned=0
2024-04-09T14:00:28.136645Z DEBUG pruner: Segment pruning started segment=Receipts purpose=User to_block=11052983 prune_mode=Before(11052984)
2024-04-09T14:00:28.136822Z DEBUG pruner: Segment pruning finished segment=Receipts purpose=User to_block=11052983 prune_mode=Before(11052984) output.pruned=0
2024-04-09T14:00:28.136827Z DEBUG pruner: Nothing to prune for the segment segment=ContractLogs purpose=User
2024-04-09T14:00:28.136848Z DEBUG pruner: Segment pruning started segment=SenderRecovery purpose=User to_block=19618541 prune_mode=Full
2024-04-09T14:00:28.146123Z DEBUG providers::db: Pruning limit reached limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 17500, deleted: 17500 }), time_limit: Some(PruneTimeLimit { limit: 100ms, start: Instant { tv_sec: 341689, tv_nsec: 920232996 } }) } deleted_entries_limit=true time_limit=false table=TransactionSenders
2024-04-09T14:00:28.146148Z DEBUG pruner: Segment pruning finished segment=SenderRecovery purpose=User to_block=19618541 prune_mode=Full output.pruned=17500
2024-04-09T14:00:28.147973Z DEBUG storage::db::mdbx: Commit total_duration=1.817931ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 2, audit: 0, write: 0, sync: 112, ending: 0, whole: 115, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0 } })) is_read_only=false
2024-04-09T14:00:28.147996Z DEBUG pruner: Pruner interrupted and has more data to prune tip_block_number=19618541 elapsed=11.475912ms deleted_entries=17500 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 17500, deleted: 17500 }), time_limit: Some(PruneTimeLimit { limit: 100ms, start: Instant { tv_sec: 341689, tv_nsec: 920232996 } }) } progress=HasMoreData(DeletedEntriesLimitReached) stats={SenderRecovery: (HasMoreData(DeletedEntriesLimitReached), 17500)}
2024-04-09T14:00:28.148064Z  INFO reth_node_core::events::node: Pruner finished tip_block_number=19618541 elapsed=11.475912ms stats={SenderRecovery: (HasMoreData(DeletedEntriesLimitReached), 17500)}
2024-04-09T14:00:28.148064Z DEBUG consensus::engine::hooks: Polled running hook with db write access hook="Prune" result=PolledHook { name: "Prune", event: Finished(Ok(())), db_access_level: ReadWrite }
2024-04-09T14:00:28.148206Z DEBUG consensus::engine::hooks: Next hook is not ready hook="StaticFile"
shekhirin commented 6 months ago

It looks like blockchain_tree::chain: Validated state root is taking too long, it's usually 100-200ms on our machines.

Can you list your hardware specs? CPU, disk, RAM

mrabino1 commented 6 months ago

intel nuc 10 i7

32GB RAM...

Model Number: Corsair MP400 Firmware Version: ECFM53.0 PCI Vendor/Subsystem ID: 0x1987 IEEE OUI Identifier: 0x6479a7 Total NVM Capacity: 4,000,787,030,016 [4.00 TB] Unallocated NVM Capacity: 0 Controller ID: 1 NVMe Version: 1.3 Number of Namespaces: 1 Namespace 1 Size/Capacity: 4,000,787,030,016 [4.00 TB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 6479a7 5b700000a1 Local Time is: Tue Apr 9 16:28:28 2024 CEST

onbjerg commented 6 months ago

Your disk is a QLC NVMe, it might be I/O? How's your disk doing during this?

mrabino1 commented 6 months ago

$ sar 5 10 Linux 6.5.0-27-generic 04/21/2024 _x8664 (12 CPU)

04:09:35 PM CPU %user %nice %system %iowait %steal %idle 04:09:40 PM all 11.95 0.00 1.53 2.48 0.00 84.04 04:09:45 PM all 9.03 0.00 1.17 0.65 0.00 89.15 04:09:50 PM all 10.63 0.00 1.39 0.80 0.00 87.18 04:09:55 PM all 12.80 0.00 1.18 0.59 0.00 85.43 04:10:00 PM all 6.69 0.00 1.12 0.52 0.00 91.67 04:10:05 PM all 12.02 0.00 2.06 2.88 0.00 83.05 04:10:10 PM all 9.24 0.00 1.04 0.90 0.00 88.81 04:10:15 PM all 10.43 0.00 2.07 5.16 0.00 82.35 04:10:20 PM all 8.76 0.00 1.22 0.82 0.00 89.20 04:10:25 PM all 7.92 0.00 0.94 1.90 0.00 89.24 Average: all 9.94 0.00 1.37 1.67 0.00 87.02

$

It is clear to me the I/O has an impact.. another almost identical node (but w/ a better I/O sync'ed far faster and doesnt have those same delays as above (all sub-500ms)... that said, it does seem like an area for efficiency gains to reduce the I/O where possible.

mrabino1 commented 5 months ago

server1 (better i/O) Apr 23 07:14:25 Elapsed: 152.05018ms Apr 23 07:14:37 Elapsed: 261.680785ms Apr 23 07:14:47 Elapsed: 37.784356ms Apr 23 07:15:01 Elapsed: 368.35482ms Apr 23 07:15:12 Elapsed: 280.652377ms Apr 23 07:15:24 Elapsed: 280.223828ms Apr 23 07:15:37 Elapsed: 244.580243ms Apr 23 07:15:49 Elapsed: 191.549118ms Apr 23 07:16:01 Elapsed: 205.356769ms Apr 23 07:16:13 Elapsed: 261.801143ms Apr 23 07:16:26 Elapsed: 244.748669ms Apr 23 07:16:38 Elapsed: 240.989382ms Apr 23 07:16:49 Elapsed: 180.999456ms Apr 23 07:17:01 Elapsed: 194.251421ms Apr 23 07:17:14 Elapsed: 311.026581ms Apr 23 07:17:25 Elapsed: 240.101565ms Apr 23 07:17:37 Elapsed: 269.648532ms Apr 23 07:17:48 Elapsed: 209.503331ms Apr 23 07:18:01 Elapsed: 235.142015ms Apr 23 07:18:13 Elapsed: 275.572254ms Apr 23 07:18:25 Elapsed: 242.43391ms Apr 23 07:18:37 Elapsed: 278.062585ms Apr 23 07:18:48 Elapsed: 221.737943ms Apr 23 07:19:01 Elapsed: 192.569149ms Apr 23 07:19:14 Elapsed: 267.416972ms Apr 23 07:19:25 Elapsed: 220.756035ms Apr 23 07:19:37 Elapsed: 256.327771ms Apr 23 07:20:02 Elapsed: 373.760014ms Apr 23 07:20:13 Elapsed: 281.554361ms Apr 23 07:20:24 Elapsed: 188.665252ms Apr 23 07:20:37 Elapsed: 155.47692ms Apr 23 07:20:48 Elapsed: 502.1312ms Apr 23 07:21:01 Elapsed: 163.416614ms Apr 23 07:21:12 Elapsed: 208.003696ms Apr 23 07:21:25 Elapsed: 412.296461ms Apr 23 07:21:36 Elapsed: 157.808811ms Apr 23 07:21:50 Elapsed: 174.690826ms Apr 23 07:22:00 Elapsed: 75.24048ms Apr 23 07:22:13 Elapsed: 394.711273ms Apr 23 07:22:25 Elapsed: 378.764735ms Apr 23 07:22:37 Elapsed: 198.90861ms Apr 23 07:22:47 Elapsed: 179.599268ms Apr 23 07:23:01 Elapsed: 273.34429ms Apr 23 07:23:14 Elapsed: 236.368341ms Apr 23 07:23:26 Elapsed: 186.958467ms Apr 23 07:23:37 Elapsed: 221.950975ms Apr 23 07:23:48 Elapsed: 222.839018ms Apr 23 07:24:00 Elapsed: 211.030149ms Apr 23 07:24:13 Elapsed: 172.855423ms Apr 23 07:24:25 Elapsed: 201.99588ms Apr 23 07:24:38 Elapsed: 1.501194002s Apr 23 07:24:49 Elapsed: 272.043364ms Apr 23 07:25:00 Elapsed: 210.979881ms Apr 23 07:25:12 Elapsed: 243.263689ms Apr 23 07:25:25 Elapsed: 294.590957ms Apr 23 07:25:36 Elapsed: 138.575991ms Apr 23 07:25:48 Elapsed: 284.643584ms Apr 23 07:26:00 Elapsed: 190.194046ms Apr 23 07:26:12 Elapsed: 390.118577ms Apr 23 07:26:25 Elapsed: 558.036119ms Apr 23 07:26:35 Elapsed: 87.875318ms Apr 23 07:26:49 Elapsed: 386.008665ms Apr 23 07:27:01 Elapsed: 192.611783ms Apr 23 07:27:13 Elapsed: 260.044119ms Apr 23 07:27:25 Elapsed: 177.921176ms Apr 23 07:27:38 Elapsed: 131.559737ms Apr 23 07:27:49 Elapsed: 808.67959ms Apr 23 07:28:01 Elapsed: 945.959087ms Apr 23 07:28:13 Elapsed: 198.224418ms Apr 23 07:28:24 Elapsed: 181.957786ms Apr 23 07:28:37 Elapsed: 399.923465ms Apr 23 07:28:49 Elapsed: 107.140283ms Apr 23 07:29:02 Elapsed: 333.654036ms Apr 23 07:29:12 Elapsed: 245.839988ms Apr 23 07:29:24 Elapsed: 86.591532ms Apr 23 07:29:38 Elapsed: 1.519810078s Apr 23 07:29:51 Elapsed: 482.035006ms Apr 23 07:30:01 Elapsed: 223.323682ms Apr 23 07:30:13 Elapsed: 409.344207ms Apr 23 07:30:40 Elapsed: 356.625761ms Apr 23 07:30:41 Elapsed: 147.354333ms Apr 23 07:30:50 Elapsed: 535.974984ms Apr 23 07:31:02 Elapsed: 452.91028ms Apr 23 07:31:13 Elapsed: 347.760663ms Apr 23 07:31:24 Elapsed: 198.136742ms Apr 23 07:31:36 Elapsed: 230.745916ms Apr 23 07:31:49 Elapsed: 134.272177ms Apr 23 07:32:02 Elapsed: 361.748671ms Apr 23 07:32:13 Elapsed: 232.568681ms Apr 23 07:32:24 Elapsed: 28.053786ms Apr 23 07:32:37 Elapsed: 447.0122ms Apr 23 07:32:49 Elapsed: 232.62966ms Apr 23 07:33:02 Elapsed: 659.473319ms Apr 23 07:33:13 Elapsed: 235.452376ms Apr 23 07:33:25 Elapsed: 246.623758ms Apr 23 07:33:36 Elapsed: 315.453918ms Apr 23 07:33:48 Elapsed: 212.262953ms Apr 23 07:34:01 Elapsed: 186.557005ms Apr 23 07:34:13 Elapsed: 284.115603ms Apr 23 07:34:25 Elapsed: 210.480052ms Apr 23 07:34:38 Elapsed: 233.104103ms Apr 23 07:34:49 Elapsed: 241.311878ms Apr 23 07:35:00 Elapsed: 69.200131ms Apr 23 07:35:13 Elapsed: 403.311903ms Apr 23 07:35:26 Elapsed: 306.154399ms Apr 23 07:35:37 Elapsed: 233.264284ms Apr 23 07:35:50 Elapsed: 287.445434ms Apr 23 07:36:00 Elapsed: 183.678163ms Apr 23 07:36:12 Elapsed: 275.35332ms Apr 23 07:36:27 Elapsed: 202.664258ms Apr 23 07:37:01 Elapsed: 193.007809ms Apr 23 07:37:13 Elapsed: 214.712132ms Apr 23 07:37:26 Elapsed: 268.747938ms Apr 23 07:37:37 Elapsed: 204.008581ms Apr 23 07:37:49 Elapsed: 238.603596ms Apr 23 07:38:01 Elapsed: 177.876593ms Apr 23 07:38:13 Elapsed: 290.41351ms Apr 23 07:38:25 Elapsed: 202.158592ms Apr 23 07:38:37 Elapsed: 390.015403ms Apr 23 07:38:49 Elapsed: 287.126597ms Apr 23 07:39:01 Elapsed: 199.999473ms Apr 23 07:39:13 Elapsed: 228.636985ms Apr 23 07:39:25 Elapsed: 242.127814ms Apr 23 07:39:37 Elapsed: 321.512995ms Apr 23 07:39:48 Elapsed: 59.641134ms Apr 23 07:39:59 Elapsed: 109.138216ms Apr 23 07:40:13 Elapsed: 363.769601ms Apr 23 07:40:24 Elapsed: 484.084731ms Apr 23 07:40:37 Elapsed: 196.919372ms Apr 23 07:40:49 Elapsed: 461.683665ms Apr 23 07:41:01 Elapsed: 153.267783ms Apr 23 07:41:13 Elapsed: 357.500798ms Apr 23 07:41:24 Elapsed: 197.705664ms Apr 23 07:41:36 Elapsed: 284.855334ms Apr 23 07:41:49 Elapsed: 197.95118ms Apr 23 07:42:01 Elapsed: 254.985564ms Apr 23 07:42:12 Elapsed: 204.325417ms Apr 23 07:42:24 Elapsed: 137.855079ms Apr 23 07:42:37 Elapsed: 275.08574ms Apr 23 07:42:52 Elapsed: 278.999337ms Apr 23 07:43:01 Elapsed: 143.184288ms Apr 23 07:43:13 Elapsed: 249.734846ms Apr 23 07:43:25 Elapsed: 212.814015ms Apr 23 07:43:37 Elapsed: 272.668885ms Apr 23 07:43:50 Elapsed: 203.464211ms Apr 23 07:44:01 Elapsed: 311.244986ms Apr 23 07:44:13 Elapsed: 202.284731ms Apr 23 07:44:24 Elapsed: 238.355791ms Apr 23 07:44:37 Elapsed: 230.38016ms Apr 23 07:44:49 Elapsed: 259.767288ms Apr 23 07:45:00 Elapsed: 185.497064ms Apr 23 07:45:12 Elapsed: 249.840433ms Apr 23 07:45:24 Elapsed: 208.652695ms Apr 23 07:45:36 Elapsed: 195.016295ms Apr 23 07:45:49 Elapsed: 272.417005ms Apr 23 07:45:59 Elapsed: 43.064735ms Apr 23 07:46:13 Elapsed: 359.14536ms Apr 23 07:46:25 Elapsed: 171.85812ms Apr 23 07:46:36 Elapsed: 217.849766ms Apr 23 07:46:49 Elapsed: 200.282419ms Apr 23 07:47:01 Elapsed: 230.906842ms Apr 23 07:47:13 Elapsed: 194.288968ms Apr 23 07:47:26 Elapsed: 217.072489ms Apr 23 07:47:37 Elapsed: 275.938509ms Apr 23 07:47:49 Elapsed: 893.806342ms Apr 23 07:48:01 Elapsed: 235.288638ms Apr 23 07:48:13 Elapsed: 309.898383ms Apr 23 07:48:24 Elapsed: 201.043034ms Apr 23 07:48:37 Elapsed: 187.194034ms Apr 23 07:48:48 Elapsed: 314.916895ms Apr 23 07:49:00 Elapsed: 142.013046ms Apr 23 07:49:13 Elapsed: 285.996958ms Apr 23 07:49:25 Elapsed: 291.755685ms Apr 23 07:49:36 Elapsed: 200.937022ms Apr 23 07:49:49 Elapsed: 228.478151ms Apr 23 07:50:01 Elapsed: 243.860866ms Apr 23 07:50:13 Elapsed: 257.953318ms Apr 23 07:50:24 Elapsed: 276.683806ms Apr 23 07:50:36 Elapsed: 335.66458ms

Server 2 Apr 23 07:00:13 Elapsed: 560.354167ms Apr 23 07:00:26 Elapsed: 1.666768897s Apr 23 07:00:38 Elapsed: 394.684706ms Apr 23 07:00:50 Elapsed: 1.610695108s Apr 23 07:01:01 Elapsed: 590.16303ms Apr 23 07:01:11 Elapsed: 224.535302ms Apr 23 07:01:26 Elapsed: 1.894779021s Apr 23 07:01:37 Elapsed: 462.427824ms Apr 23 07:01:50 Elapsed: 1.547277229s Apr 23 07:02:01 Elapsed: 326.340503ms Apr 23 07:02:14 Elapsed: 1.711266248s Apr 23 07:02:25 Elapsed: 310.983708ms Apr 23 07:02:38 Elapsed: 1.433023906s Apr 23 07:02:48 Elapsed: 381.999328ms Apr 23 07:03:02 Elapsed: 1.489969109s Apr 23 07:03:13 Elapsed: 345.238821ms Apr 23 07:03:26 Elapsed: 1.401196335s Apr 23 07:03:36 Elapsed: 250.730744ms Apr 23 07:03:48 Elapsed: 298.546579ms Apr 23 07:04:02 Elapsed: 339.576048ms Apr 23 07:04:14 Elapsed: 425.516111ms Apr 23 07:04:25 Elapsed: 70.878115ms Apr 23 07:04:38 Elapsed: 1.034398963s Apr 23 07:04:49 Elapsed: 481.238203ms Apr 23 07:05:01 Elapsed: 601.063773ms Apr 23 07:05:12 Elapsed: 662.701495ms Apr 23 07:05:25 Elapsed: 412.100601ms Apr 23 07:05:37 Elapsed: 520.355659ms Apr 23 07:05:49 Elapsed: 449.917473ms Apr 23 07:06:02 Elapsed: 366.660233ms Apr 23 07:06:14 Elapsed: 583.366947ms Apr 23 07:06:26 Elapsed: 583.950877ms Apr 23 07:06:37 Elapsed: 510.529738ms Apr 23 07:06:50 Elapsed: 431.335121ms Apr 23 07:07:01 Elapsed: 348.812264ms Apr 23 07:07:14 Elapsed: 839.450636ms Apr 23 07:07:24 Elapsed: 146.68422ms Apr 23 07:07:37 Elapsed: 941.117239ms Apr 23 07:07:49 Elapsed: 402.780697ms Apr 23 07:08:01 Elapsed: 397.395379ms Apr 23 07:08:13 Elapsed: 595.621689ms Apr 23 07:08:25 Elapsed: 412.952004ms Apr 23 07:08:36 Elapsed: 297.998509ms Apr 23 07:08:48 Elapsed: 59.184692ms Apr 23 07:09:01 Elapsed: 685.032652ms Apr 23 07:09:13 Elapsed: 480.24434ms Apr 23 07:09:25 Elapsed: 157.256718ms Apr 23 07:09:37 Elapsed: 702.282253ms Apr 23 07:09:50 Elapsed: 387.083375ms Apr 23 07:10:01 Elapsed: 380.849743ms Apr 23 07:10:13 Elapsed: 438.433305ms Apr 23 07:10:25 Elapsed: 610.364546ms Apr 23 07:10:36 Elapsed: 339.465104ms Apr 23 07:10:48 Elapsed: 84.91788ms Apr 23 07:11:01 Elapsed: 966.234988ms Apr 23 07:11:13 Elapsed: 521.671098ms Apr 23 07:11:24 Elapsed: 470.771844ms Apr 23 07:11:37 Elapsed: 496.29256ms Apr 23 07:11:49 Elapsed: 370.687021ms Apr 23 07:12:01 Elapsed: 513.399308ms Apr 23 07:12:12 Elapsed: 520.558675ms Apr 23 07:12:25 Elapsed: 378.247939ms Apr 23 07:12:36 Elapsed: 441.909569ms Apr 23 07:12:48 Elapsed: 441.084542ms Apr 23 07:13:01 Elapsed: 365.882591ms Apr 23 07:13:13 Elapsed: 540.100402ms Apr 23 07:13:24 Elapsed: 223.52483ms Apr 23 07:13:37 Elapsed: 482.750434ms Apr 23 07:13:50 Elapsed: 370.049388ms Apr 23 07:14:01 Elapsed: 372.079771ms Apr 23 07:14:15 Elapsed: 1.709348204s Apr 23 07:14:25 Elapsed: 232.995848ms Apr 23 07:14:37 Elapsed: 468.655437ms Apr 23 07:14:47 Elapsed: 42.355785ms Apr 23 07:15:01 Elapsed: 587.248861ms Apr 23 07:15:12 Elapsed: 535.699348ms Apr 23 07:15:24 Elapsed: 471.772235ms Apr 23 07:15:37 Elapsed: 370.923779ms Apr 23 07:15:49 Elapsed: 274.881807ms Apr 23 07:16:01 Elapsed: 411.438742ms Apr 23 07:16:13 Elapsed: 418.513015ms Apr 23 07:16:25 Elapsed: 404.989149ms Apr 23 07:16:38 Elapsed: 436.714135ms Apr 23 07:16:49 Elapsed: 252.724988ms Apr 23 07:17:01 Elapsed: 300.226795ms Apr 23 07:17:13 Elapsed: 446.539401ms Apr 23 07:17:25 Elapsed: 513.984856ms Apr 23 07:17:37 Elapsed: 552.036371ms Apr 23 07:17:48 Elapsed: 374.747344ms Apr 23 07:18:01 Elapsed: 419.313696ms Apr 23 07:18:13 Elapsed: 510.453796ms Apr 23 07:18:25 Elapsed: 400.070697ms Apr 23 07:18:37 Elapsed: 670.121796ms Apr 23 07:18:48 Elapsed: 366.540793ms Apr 23 07:19:01 Elapsed: 407.42524ms Apr 23 07:19:14 Elapsed: 513.720076ms Apr 23 07:19:25 Elapsed: 389.163218ms Apr 23 07:19:37 Elapsed: 470.477108ms Apr 23 07:20:02 Elapsed: 657.265746ms Apr 23 07:20:13 Elapsed: 535.065259ms Apr 23 07:20:24 Elapsed: 391.289473ms Apr 23 07:20:37 Elapsed: 318.334109ms Apr 23 07:20:49 Elapsed: 887.40494ms Apr 23 07:21:01 Elapsed: 285.359178ms Apr 23 07:21:12 Elapsed: 390.762709ms Apr 23 07:21:25 Elapsed: 736.730152ms Apr 23 07:21:37 Elapsed: 369.863516ms Apr 23 07:21:50 Elapsed: 329.09485ms Apr 23 07:22:00 Elapsed: 178.565834ms Apr 23 07:22:13 Elapsed: 814.055736ms Apr 23 07:22:25 Elapsed: 678.577347ms Apr 23 07:22:37 Elapsed: 298.683359ms Apr 23 07:22:48 Elapsed: 282.841973ms Apr 23 07:23:02 Elapsed: 611.34504ms Apr 23 07:23:14 Elapsed: 408.291973ms Apr 23 07:23:26 Elapsed: 329.424166ms Apr 23 07:23:37 Elapsed: 413.512474ms Apr 23 07:23:49 Elapsed: 397.720607ms Apr 23 07:24:00 Elapsed: 384.265069ms Apr 23 07:24:13 Elapsed: 361.670886ms Apr 23 07:24:25 Elapsed: 319.334368ms Apr 23 07:24:39 Elapsed: 2.417036913s Apr 23 07:24:49 Elapsed: 443.232482ms Apr 23 07:25:01 Elapsed: 408.085519ms Apr 23 07:25:12 Elapsed: 410.508539ms Apr 23 07:25:26 Elapsed: 549.664589ms Apr 23 07:25:36 Elapsed: 207.479969ms Apr 23 07:25:48 Elapsed: 483.652767ms Apr 23 07:26:00 Elapsed: 341.307396ms Apr 23 07:26:13 Elapsed: 557.108587ms Apr 23 07:26:25 Elapsed: 871.556083ms Apr 23 07:26:35 Elapsed: 158.262929ms Apr 23 07:26:49 Elapsed: 596.572889ms Apr 23 07:27:01 Elapsed: 266.17361ms Apr 23 07:27:13 Elapsed: 428.055238ms Apr 23 07:27:25 Elapsed: 246.163298ms Apr 23 07:27:38 Elapsed: 208.430974ms Apr 23 07:27:49 Elapsed: 1.128461162s Apr 23 07:28:01 Elapsed: 1.328813036s Apr 23 07:28:14 Elapsed: 306.120028ms Apr 23 07:28:24 Elapsed: 295.774926ms Apr 23 07:28:37 Elapsed: 595.838611ms Apr 23 07:28:49 Elapsed: 174.336266ms Apr 23 07:29:02 Elapsed: 520.13443ms Apr 23 07:29:13 Elapsed: 371.16441ms Apr 23 07:29:24 Elapsed: 165.730394ms Apr 23 07:29:39 Elapsed: 2.117952471s Apr 23 07:29:51 Elapsed: 982.636475ms Apr 23 07:30:01 Elapsed: 313.527796ms Apr 23 07:30:13 Elapsed: 770.281609ms Apr 23 07:30:25 Elapsed: 466.986663ms Apr 23 07:30:36 Elapsed: 196.442638ms Apr 23 07:30:50 Elapsed: 910.72756ms Apr 23 07:31:02 Elapsed: 725.515112ms Apr 23 07:31:14 Elapsed: 635.672977ms Apr 23 07:31:24 Elapsed: 283.537012ms Apr 23 07:31:36 Elapsed: 423.575077ms Apr 23 07:31:49 Elapsed: 215.076001ms Apr 23 07:32:07 Elapsed: 4.967260448s Apr 23 07:32:18 Elapsed: 5.319927674s Apr 23 07:32:24 Elapsed: 78.691019ms Apr 23 07:32:37 Elapsed: 828.779033ms Apr 23 07:32:49 Elapsed: 517.928348ms Apr 23 07:33:02 Elapsed: 1.330886753s Apr 23 07:33:13 Elapsed: 454.141455ms Apr 23 07:33:25 Elapsed: 574.038629ms Apr 23 07:33:36 Elapsed: 679.600592ms Apr 23 07:33:48 Elapsed: 342.704987ms Apr 23 07:34:01 Elapsed: 339.432972ms Apr 23 07:34:13 Elapsed: 636.446696ms Apr 23 07:34:25 Elapsed: 347.789376ms Apr 23 07:34:38 Elapsed: 364.60045ms Apr 23 07:34:49 Elapsed: 415.083795ms Apr 23 07:35:00 Elapsed: 100.026439ms Apr 23 07:35:13 Elapsed: 690.798604ms Apr 23 07:35:26 Elapsed: 555.633078ms Apr 23 07:35:37 Elapsed: 382.584819ms Apr 23 07:35:50 Elapsed: 580.425554ms Apr 23 07:36:00 Elapsed: 313.832843ms Apr 23 07:36:12 Elapsed: 420.558787ms Apr 23 07:36:27 Elapsed: 321.435002ms Apr 23 07:37:01 Elapsed: 405.18019ms Apr 23 07:37:13 Elapsed: 454.575673ms Apr 23 07:37:26 Elapsed: 635.210882ms Apr 23 07:37:37 Elapsed: 540.161092ms Apr 23 07:37:49 Elapsed: 468.289779ms Apr 23 07:38:01 Elapsed: 429.939709ms Apr 23 07:38:13 Elapsed: 598.369974ms Apr 23 07:38:25 Elapsed: 381.290312ms Apr 23 07:38:37 Elapsed: 711.669278ms Apr 23 07:38:49 Elapsed: 513.593324ms Apr 23 07:39:00 Elapsed: 304.950998ms Apr 23 07:39:13 Elapsed: 405.694373ms Apr 23 07:39:25 Elapsed: 412.995357ms Apr 23 07:39:37 Elapsed: 569.773954ms Apr 23 07:39:48 Elapsed: 113.882884ms Apr 23 07:39:59 Elapsed: 143.034582ms Apr 23 07:40:13 Elapsed: 750.521558ms Apr 23 07:40:24 Elapsed: 813.279497ms Apr 23 07:40:37 Elapsed: 359.839876ms Apr 23 07:40:49 Elapsed: 784.916452ms Apr 23 07:41:01 Elapsed: 259.199552ms Apr 23 07:41:13 Elapsed: 584.004182ms Apr 23 07:41:24 Elapsed: 377.814606ms Apr 23 07:41:37 Elapsed: 477.972902ms Apr 23 07:41:49 Elapsed: 365.802341ms Apr 23 07:42:01 Elapsed: 475.893066ms Apr 23 07:42:12 Elapsed: 312.489786ms Apr 23 07:42:24 Elapsed: 221.562872ms Apr 23 07:42:37 Elapsed: 574.154649ms Apr 23 07:42:52 Elapsed: 532.791659ms Apr 23 07:43:01 Elapsed: 322.374585ms Apr 23 07:43:13 Elapsed: 534.095795ms Apr 23 07:43:25 Elapsed: 466.015692ms Apr 23 07:43:37 Elapsed: 542.456041ms Apr 23 07:43:50 Elapsed: 454.47732ms Apr 23 07:44:01 Elapsed: 498.816731ms Apr 23 07:44:13 Elapsed: 416.187298ms Apr 23 07:44:25 Elapsed: 432.404687ms Apr 23 07:44:37 Elapsed: 454.77717ms Apr 23 07:44:49 Elapsed: 498.383135ms Apr 23 07:45:00 Elapsed: 355.147476ms Apr 23 07:45:13 Elapsed: 665.405788ms Apr 23 07:45:24 Elapsed: 489.447493ms Apr 23 07:45:37 Elapsed: 485.869677ms Apr 23 07:45:50 Elapsed: 516.767755ms Apr 23 07:45:59 Elapsed: 68.860666ms Apr 23 07:46:13 Elapsed: 680.778403ms Apr 23 07:46:25 Elapsed: 303.443227ms Apr 23 07:46:36 Elapsed: 402.410394ms Apr 23 07:46:49 Elapsed: 379.38305ms Apr 23 07:47:01 Elapsed: 381.805591ms Apr 23 07:47:13 Elapsed: 438.868958ms Apr 23 07:47:26 Elapsed: 358.458964ms Apr 23 07:47:37 Elapsed: 491.432886ms Apr 23 07:47:51 Elapsed: 2.126578553s Apr 23 07:48:01 Elapsed: 415.400718ms Apr 23 07:48:13 Elapsed: 538.094695ms Apr 23 07:48:24 Elapsed: 317.001784ms Apr 23 07:48:37 Elapsed: 302.128376ms Apr 23 07:48:49 Elapsed: 514.737086ms Apr 23 07:49:00 Elapsed: 210.190977ms Apr 23 07:49:13 Elapsed: 588.251888ms Apr 23 07:49:25 Elapsed: 593.195588ms Apr 23 07:49:37 Elapsed: 348.621679ms

beta.6 was upgraded about 30min ago.. and appears to be helping.. you put them side by side.. you can see the performance diff between the two.. disk i/o is the only explanation.. but beta.6 appears to be helping !

github-actions[bot] commented 5 months ago

This issue is stale because it has been open for 21 days with no activity.

github-actions[bot] commented 4 months ago

This issue is stale because it has been open for 21 days with no activity.

github-actions[bot] commented 4 months ago

This issue was closed because it has been inactive for 7 days since being marked as stale.

github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 21 days with no activity.

github-actions[bot] commented 2 months ago

This issue was closed because it has been inactive for 7 days since being marked as stale.