search

hyperledger / besu

An enterprise-grade Java-based, Apache 2.0 licensed Ethereum client https://wiki.hyperledger.org/display/besu
https://www.hyperledger.org/projects/besu
Apache License 2.0
1.49k stars 812 forks source link

Threads blocked followed by crash #5043

Closed stefa2k closed 1 year ago

stefa2k commented 1 year ago

Description

We are running a Besu/Nimbus staking node and noticed a crash last night.

Expected behavior: No crash.

Actual behavior: This error is all over the logs for something like 4 hours:

{"log":"2023-02-03 04:21:49.766+00:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-1,5,main] has been blocked for 400098 ms, time limit is 60000 ms\n","stream":"stdout","time":"2023-02-03T04:22:13.749539938Z"}
{"log":"io.vertx.core.VertxException: Thread blocked\n","stream":"stdout","time":"2023-02-03T04:22:13.749590402Z"}
{"log":"\u0009at app//io.prometheus.client.CKMSQuantiles.\u003cinit\u003e(CKMSQuantiles.java:71)\n","stream":"stdout","time":"2023-02-03T04:22:13.749595422Z"}
{"log":"\u0009at app//io.prometheus.client.TimeWindowQuantiles.rotate(TimeWindowQuantiles.java:45)\n","stream":"stdout","time":"2023-02-03T04:22:13.74959981Z"}
{"log":"\u0009at app//io.prometheus.client.TimeWindowQuantiles.get(TimeWindowQuantiles.java:31)\n","stream":"stdout","time":"2023-02-03T04:22:13.749603497Z"}
{"log":"\u0009at app//io.prometheus.client.Summary$Child$Value.snapshot(Summary.java:252)\n","stream":"stdout","time":"2023-02-03T04:22:13.749607134Z"}
{"log":"\u0009at app//io.prometheus.client.Summary$Child$Value.\u003cinit\u003e(Summary.java:246)\n","stream":"stdout","time":"2023-02-03T04:22:13.749610761Z"}
{"log":"\u0009at app//io.prometheus.client.Summary$Child$Value.\u003cinit\u003e(Summary.java:238)\n","stream":"stdout","time":"2023-02-03T04:22:13.749614467Z"}
{"log":"\u0009at app//io.prometheus.client.Summary$Child.get(Summary.java:300)\n","stream":"stdout","time":"2023-02-03T04:22:13.749617995Z"}
{"log":"\u0009at app//io.prometheus.client.Summary.collect(Summary.java:353)\n","stream":"stdout","time":"2023-02-03T04:22:13.74962147Z"}
{"log":"\u0009at app//io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:190)\n","stream":"stdout","time":"2023-02-03T04:22:13.749624887Z"}
{"log":"\u0009at app//io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:223)\n","stream":"stdout","time":"2023-02-03T04:22:13.749628475Z"}
{"log":"\u0009at app//io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:144)\n","stream":"stdout","time":"2023-02-03T04:22:13.74963194Z"}
{"log":"\u0009at app//io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)\n","stream":"stdout","time":"2023-02-03T04:22:13.749635397Z"}
{"log":"\u0009at app//org.hyperledger.besu.metrics.prometheus.MetricsHttpService.lambda$metricsRequest$4(MetricsHttpService.java:205)\n","stream":"stdout","time":"2023-02-03T04:22:13.749638833Z"}
{"log":"\u0009at app//org.hyperledger.besu.metrics.prometheus.MetricsHttpService$$Lambda$1363/0x00000008406ff840.handle(Unknown Source)\n","stream":"stdout","time":"2023-02-03T04:22:13.74964237Z"}
{"log":"\u0009at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)\n","stream":"stdout","time":"2023-02-03T04:22:13.749646327Z"}
{"log":"\u0009at app//io.vertx.core.impl.ContextImpl$$Lambda$1026/0x000000084060b840.handle(Unknown Source)\n","stream":"stdout","time":"2023-02-03T04:22:13.749649824Z"}
{"log":"\u0009at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)\n","stream":"stdout","time":"2023-02-03T04:22:13.74965801Z"}
{"log":"\u0009at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)\n","stream":"stdout","time":"2023-02-03T04:22:13.749661716Z"}
{"log":"\u0009at app//io.vertx.core.impl.ContextImpl$$Lambda$1025/0x000000084060b440.run(Unknown Source)\n","stream":"stdout","time":"2023-02-03T04:22:13.749665233Z"}
{"log":"\u0009at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n","stream":"stdout","time":"2023-02-03T04:22:13.749670132Z"}
{"log":"\u0009at java.base@11.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n","stream":"stdout","time":"2023-02-03T04:22:13.749694137Z"}
{"log":"\u0009at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n","stream":"stdout","time":"2023-02-03T04:22:13.749700258Z"}

While this error was present the full node stopped being able to attest. Nimbus is used as the consensus client.

And at the end Besu crashed:

{"log":"\u0009at java.base@11.0.17/java.lang.Thread.run(Thread.java:829)\n","stream":"stdout","time":"2023-02-03T04:22:13.749705409Z"}
{"log":"2023-02-03 04:22:29.824+00:00 | vert.x-eventloop-thread-6 | WARN  | EngineQosTimer | Execution engine not called in 120 seconds, consensus client may not be connected\n","stream":"stdout","time":"2023-02-03T04:22:29.824306008Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.01389385Z"}
{"log":"# A fatal error has been detected by the Java Runtime Environment:\n","stream":"stdout","time":"2023-02-03T04:22:34.01394734Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.01395216Z"}
{"log":"#  SIGSEGV (0xb) at pc=0x0000000000000000, pid=1, tid=278364\n","stream":"stdout","time":"2023-02-03T04:22:34.013955716Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.013959344Z"}
{"log":"# JRE version: OpenJDK Runtime Environment (11.0.17+8) (build 11.0.17+8-post-Ubuntu-1ubuntu220.04)\n","stream":"stdout","time":"2023-02-03T04:22:34.01396289Z"}
{"log":"# Java VM: OpenJDK 64-Bit Server VM (11.0.17+8-post-Ubuntu-1ubuntu220.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)\n","stream":"stdout","time":"2023-02-03T04:22:34.013966657Z"}
{"log":"# Problematic frame:\n","stream":"stdout","time":"2023-02-03T04:22:34.013970193Z"}
{"log":"# C  0x0000000000000000\n","stream":"stdout","time":"2023-02-03T04:22:34.039667746Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.039704987Z"}
{"log":"# Core dump will be written. Default location: Core dumps may be processed with \"/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E\" (or dumping to /opt/besu/core.1)\n","stream":"stdout","time":"2023-02-03T04:22:34.039711499Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.039716548Z"}
{"log":"# An error report file with more information is saved as:\n","stream":"stdout","time":"2023-02-03T04:22:34.041879863Z"}
{"log":"# /tmp/hs_err_pid1.log\n","stream":"stdout","time":"2023-02-03T04:22:34.041925799Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.129292854Z"}
{"log":"# If you would like to submit a bug report, please visit:\n","stream":"stdout","time":"2023-02-03T04:22:34.129335283Z"}
{"log":"#   https://bugs.launchpad.net/ubuntu/+source/openjdk-lts\n","stream":"stdout","time":"2023-02-03T04:22:34.129341214Z"}
{"log":"# The crash happened outside the Java Virtual Machine in native code.\n","stream":"stdout","time":"2023-02-03T04:22:34.129345753Z"}
{"log":"# See problematic frame for where to report the bug.\n","stream":"stdout","time":"2023-02-03T04:22:34.12935001Z"}
{"log":"#\n","stream":"stdout","time":"2023-02-03T04:22:34.129354369Z"}
{"log":"\n","stream":"stdout","time":"2023-02-03T04:22:34.13261533Z"}
{"log":"[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f7a038f8941]\n","stream":"stdout","time":"2023-02-03T04:22:34.132634957Z"}
{"log":"\n","stream":"stdout","time":"2023-02-03T04:22:34.132639886Z"}
{"log":"Setting logging level to INFO\n","stream":"stdout","time":"2023-02-03T04:22:38.812814314Z"}

Frequency: Saw this the first time.

Versions (Add all that apply)

Additional Information (Add any of the following or anything else that may be relevant)

docker inspect:

        "Config": {
            "Env": [
                "JAVA_OPTS=-Xmx4g",
                "PATH=/opt/besu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "BESU_RPC_HTTP_HOST=0.0.0.0",
                "BESU_RPC_WS_HOST=0.0.0.0",
                "BESU_GRAPHQL_HTTP_HOST=0.0.0.0",
                "BESU_PID_PATH=/tmp/pid",
                "OTEL_RESOURCE_ATTRIBUTES=service.name=besu,service.version=22.10.3",
                "OLDPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--pruning-enabled=true",
                "--network=mainnet",
                "--data-path=/opt/app/data",
                "--data-storage-format=BONSAI",
                "--nat-method=AUTO",
                "--p2p-port=xxxxx",
                "--p2p-host=x.x.x.x",
                "--rpc-http-enabled=true",
                "--rpc-http-host=0.0.0.0",
                "--rpc-http-cors-origins=*",
                "--rpc-http-port=8545",
                "--rpc-http-api=ETH,NET,WEB3,ADMIN",
                "--rpc-ws-enabled=true",
                "--rpc-ws-host=0.0.0.0",
                "--rpc-ws-port=8546",
                "--host-allowlist=*",
                "--metrics-enabled",
                "--metrics-host=0.0.0.0",
                "--metrics-port=9545",
                "--logging=INFO",
                "--engine-rpc-enabled=true",
                "--engine-host-allowlist=*",
                "--engine-rpc-port=8551",
                "--engine-jwt-enabled=true",
                "--engine-jwt-secret=/engine.jwt"
            ],
ibhagwan commented 1 year ago

May or may not be related, I am experiencing blocked threads since upgrading to 23.1.0, as it breaks this is what the log shows:

2023-02-20 22:31:00.874-05:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0x1f8e929c7fbe1e0e935968198b82e2347cc48e236df11c0e6783c1220e91fe57, finalized: 0xd1f65a616b257b0fd70f382ba03c17425c157e844841f0416206a5145f2af222, safeBlockHash: 0x94e592eb2d10e3f275cc380590f2dd9991d249576451ce158bac688bb1b7b509
2023-02-20 22:31:12.896-05:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #16,674,223 / 156 tx / base fee 23.40 gwei / 11,651,140 (38.8%) gas / (0x548bf21aa77753f05723ccf436a4e8c87d8b4162e2c686143eaf4daf7e2ed3fd) in 0.388s. Peers: 100
2023-02-20 22:31:25.419-05:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #16,674,224 / 121 tx / base fee 22.75 gwei / 17,757,590 (59.2%) gas / (0xc91dbad029f24c097ff35f0368106eee6b2d0449ef81806f2c3ab394f1731af2) in 0.672s. Peers: 100
2023-02-20 22:33:26.307-05:00 | vert.x-eventloop-thread-1 | WARN  | EngineQosTimer | Execution engine not called in 120 seconds, consensus client may not be connected
2023-02-20 22:33:29.693-05:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #16,674,225 / 143 tx / base fee 23.27 gwei / 13,620,351 (45.4%) gas / (0xac889d77406552e54c1ffd30560b3d7434a30ffa3e1eb9d15d7961ab5bc3bd63) in 0.733s. Peers: 93
2023-02-20 22:33:30.512-05:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #16,674,226 / 89 tx / base fee 23.01 gwei / 10,894,362 (36.3%) gas / (0xb253635d1f7d3991c05603b036d03d81406c6b71d9a16d2b80ec892006f0e81d) in 0.560s. Peers: 93
2023-02-20 22:33:30.590-05:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xac889d77406552e54c1ffd30560b3d7434a30ffa3e1eb9d15d7961ab5bc3bd63, finalized: 0xd1f65a616b257b0fd70f382ba03c17425c157e844841f0416206a5145f2af222, safeBlockHash: 0x94e592eb2d10e3f275cc380590f2dd9991d249576451ce158bac688bb1b7b509
2023-02-20 22:34:31.060-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 60317 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at app//org.rocksdb.Transaction.get(Native Method)
        at app//org.rocksdb.Transaction.get(Transaction.java:278)
        at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBSnapshotTransaction.get(RocksDBSnapshotTransaction.java:103)
        at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueSnapshot.get(RocksDBColumnarKeyValueSnapshot.java:65)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage.getAccount(BonsaiWorldStateKeyValueStorage.java:88)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.get(BonsaiPersistedWorldState.java:407)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.loadAccount(BonsaiWorldStateUpdater.java:165)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getForMutation(BonsaiWorldStateUpdater.java:156)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getForMutation(BonsaiWorldStateUpdater.java:51)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:113)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.get(BonsaiWorldStateUpdater.java:101)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.operation.StaticCallOperation.cost(StaticCallOperation.java:100)
        at app//org.hyperledger.besu.evm.operation.AbstractCallOperation.execute(AbstractCallOperation.java:167)
        at app//org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:307)
        at app//org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
        at app//org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
        at app//org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:495)
        at app//org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:405)
        at app//org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:110)
        at app//org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
        at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:218)
        at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:477)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:491)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:235)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod$$Lambda$1623/0x00000008012e2160.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at app//io.vertx.core.impl.TaskQueue$$Lambda$763/0x0000000801089ee8.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
2023-02-20 22:34:31.612-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-2,5,main] has been blocked for 60845 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeRequest(JsonRpcExecutorHandler.java:118)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeJsonObjectRequest(JsonRpcExecutorHandler.java:131)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:78)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1178/0x000000080119fc40.handle(Unknown Source)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1561/0x00000008012c3cd8.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
2023-02-20 22:34:31.612-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-6,5,main] has been blocked for 60870 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeRequest(JsonRpcExecutorHandler.java:118)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeJsonObjectRequest(JsonRpcExecutorHandler.java:131)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:78)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1178/0x000000080119fc40.handle(Unknown Source)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1561/0x00000008012c3cd8.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
2023-02-20 22:34:32.059-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 61317 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at app//org.rocksdb.Transaction.get(Native Method)
        at app//org.rocksdb.Transaction.get(Transaction.java:278)
        at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBSnapshotTransaction.get(RocksDBSnapshotTransaction.java:103)
        at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueSnapshot.get(RocksDBColumnarKeyValueSnapshot.java:65)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage.getAccount(BonsaiWorldStateKeyValueStorage.java:88)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.get(BonsaiPersistedWorldState.java:407)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.loadAccount(BonsaiWorldStateUpdater.java:165)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getForMutation(BonsaiWorldStateUpdater.java:156)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getForMutation(BonsaiWorldStateUpdater.java:51)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:113)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.get(BonsaiWorldStateUpdater.java:101)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.operation.StaticCallOperation.cost(StaticCallOperation.java:100)
        at app//org.hyperledger.besu.evm.operation.AbstractCallOperation.execute(AbstractCallOperation.java:167)
        at app//org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:307)
        at app//org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
        at app//org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
        at app//org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:495)
        at app//org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:405)
        at app//org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:110)
        at app//org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
        at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:218)
        at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:477)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:491)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:235)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod$$Lambda$1623/0x00000008012e2160.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at app//io.vertx.core.impl.TaskQueue$$Lambda$763/0x0000000801089ee8.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
2023-02-20 22:34:32.612-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-2,5,main] has been blocked for 61845 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeRequest(JsonRpcExecutorHandler.java:118)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeJsonObjectRequest(JsonRpcExecutorHandler.java:131)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:78)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1178/0x000000080119fc40.handle(Unknown Source)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1561/0x00000008012c3cd8.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
2023-02-20 22:34:32.612-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-6,5,main] has been blocked for 61870 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at java.base@17.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@17.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)
        at java.base@17.0.5/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
        at java.base@17.0.5/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeRequest(JsonRpcExecutorHandler.java:118)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.executeJsonObjectRequest(JsonRpcExecutorHandler.java:131)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:78)
        at app//org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler$$Lambda$1178/0x000000080119fc40.handle(Unknown Source)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
        at app//io.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$1561/0x00000008012c3cd8.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)
2023-02-20 22:34:33.059-05:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-worker-thread-0,5,main] has been blocked for 62317 ms, time limit is 60000 ms
io.vertx.core.VertxException: Thread blocked
        at app//org.rocksdb.Transaction.get(Native Method)
        at app//org.rocksdb.Transaction.get(Transaction.java:278)
        at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBSnapshotTransaction.get(RocksDBSnapshotTransaction.java:103)
        at app//org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueSnapshot.get(RocksDBColumnarKeyValueSnapshot.java:65)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateKeyValueStorage.getAccount(BonsaiWorldStateKeyValueStorage.java:88)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiPersistedWorldState.get(BonsaiPersistedWorldState.java:407)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.loadAccount(BonsaiWorldStateUpdater.java:165)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getForMutation(BonsaiWorldStateUpdater.java:156)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.getForMutation(BonsaiWorldStateUpdater.java:51)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:113)
        at app//org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateUpdater.get(BonsaiWorldStateUpdater.java:101)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:98)
        at app//org.hyperledger.besu.evm.operation.StaticCallOperation.cost(StaticCallOperation.java:100)
        at app//org.hyperledger.besu.evm.operation.AbstractCallOperation.execute(AbstractCallOperation.java:167)
        at app//org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:307)
        at app//org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
        at app//org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
        at app//org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:495)
        at app//org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:405)
        at app//org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:110)
        at app//org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
        at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:218)
        at app//org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:135)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:477)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:491)
        at app//org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:235)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
        at app//org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod$$Lambda$1623/0x00000008012e2160.handle(Unknown Source)
        at app//io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1405/0x00000008012438d0.handle(Unknown Source)
        at app//io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
        at app//io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
        at app//io.vertx.core.impl.ContextImpl$$Lambda$1402/0x0000000801242e18.run(Unknown Source)
        at app//io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at app//io.vertx.core.impl.TaskQueue$$Lambda$763/0x0000000801089ee8.run(Unknown Source)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base@17.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base@17.0.5/java.lang.Thread.run(Thread.java:833)

As opposed to what OP is describing I don't get to a crash, besu just stays in the blocked thread loop until I kill the process and restart.

ahamlat commented 1 year ago

@stefa2k I see in the logs 

{"log":"2023-02-03 04:22:29.824+00:00 | vert.x-eventloop-thread-6 | WARN  | EngineQosTimer | Execution engine not called in 120 seconds, consensus client may not be connected\n","stream":"stdout","time":"2023-02-03T04:22:29.824306008Z"}

Was you CL running correctly ? Could you check the logs if you still have them, otherwise share if you still have the issue Please, share also the content of this log /tmp/hs_err_pid1.log @ibhagwan Could you share your hardware setup, besu options and your CL logs ?

ibhagwan commented 1 year ago

@stefa2k I see in the logs

{"log":"2023-02-03 04:22:29.824+00:00 | vert.x-eventloop-thread-6 | WARN  | EngineQosTimer | Execution engine not called in 120 seconds, consensus client may not be connected\n","stream":"stdout","time":"2023-02-03T04:22:29.824306008Z"}

Was you CL running correctly ? Could you check the logs if you still have them, otherwise share if you still have the issue Please, share also the content of this log /tmp/hs_err_pid1.log @ibhagwan Could you share your hardware setup, besu options and your CL logs ?

CL is running what seems to be perfect at all times, before this issue started to appear daily with 23.1.0 I wouldn’t restart the CL (nimbus) for month, the logs are normal with the usual messages.

Unfortunately I don’t have the nimbus log from the OP (which you are referring to in this comment), I do have the logs of yesterday still and will probably have the issue again sometime again, although I now downgraded back to 22.10.3. I mixed up this issue with #4904 which I opened, I’ll look for the corresponding logs and post what I find.

stefa2k commented 1 year ago

@ahamlat Thanks for the response, hard to tell about the logs, they already rotated away. On this particular node Nimbus was used as the consensus client. We run also other Besu nodes but didn't observe something similar on them.

I'll report here as soon as we get this again with the logs you requested.

non-fungible-nelson commented 1 year ago

Hi @stefa2k any status here?

Taschentuch9 commented 1 year ago

I just had the same issue caused by a blocked thread. I am running besu (23.1.1) and nimbus (23.3.0) on Goerli.

My CPU suddenly started running on 100%. I stopped and restarzed besu and after a backward sync everything was fine again.

kooshan75 commented 1 year ago

I just had the same issue caused by a blocked thread. I am running besu (23.1.1) and nimbus (23.3.0) on Goerli.

My CPU suddenly started running on 100%. I stopped and restarzed besu and after a backward sync everything was fine again.

Same issue with besu 23.1.2 latest atm and lighthouse.

smasher816 commented 1 year ago

Same issue with 23.1.3 and nimbus

Apr 24 17:08:25 hoenir besu[154406]: 2023-04-24 17:08:25.298-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0x3bcca6674d1edc983e6fcfa90c693fcc07553c13e14f3d50dbc95ecd75b6f6f3, finalized: 0x348d03191b0f28681e36ad35a862e9db97242debbac8aa71ae374d97b3caf8d4, safeBlockHash: 0xad0ea7f6bf6d2273206a21ba0d391f127a84989740f2b085d68d4f0a39d8930a
Apr 24 17:08:38 hoenir besu[154406]: 2023-04-24 17:08:38.074-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,614 / 149 tx / 16 ws / base fee 33.63 gwei / 18,559,792 (61.9%) gas / (0x7d586cee47a2ad41d29e86b6b6ba55ef7a8ade5b01fea19171dc1d569b425b7b) in 0.512s. Peers: 25
Apr 24 17:08:49 hoenir besu[154406]: 2023-04-24 17:08:49.627-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,615 / 93 tx / 16 ws / base fee 34.63 gwei / 9,729,956 (32.4%) gas / (0x37429d1c18d1f3cf7ebb3d121b19776eb48c54f6900df35000f7aa0949850dcf) in 0.386s. Peers: 25
Apr 24 17:09:02 hoenir besu[154406]: 2023-04-24 17:09:02.562-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,616 / 161 tx / 16 ws / base fee 33.11 gwei / 15,492,038 (51.6%) gas / (0xa0c419d709d8adf3f49186747d3effb8efeb1c02891486e138117e8edb5225b8) in 0.593s. Peers: 25
Apr 24 17:09:14 hoenir besu[154406]: 2023-04-24 17:09:14.401-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,617 / 114 tx / 16 ws / base fee 33.24 gwei / 13,225,251 (44.1%) gas / (0x25436ad89846c369c7f822bf8bde6b97eb83be51249c1bb1eac7010bbc812991) in 0.497s. Peers: 25
Apr 24 17:09:24 hoenir besu[154406]: 2023-04-24 17:09:24.746-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,618 / 140 tx / 16 ws / base fee 32.75 gwei / 12,613,479 (42.0%) gas / (0x81941bb2453622b520f4ad2a32174efcf684228ed1c0eb01c0ad40c1edd29e88) in 0.427s. Peers: 25
Apr 24 17:11:20 hoenir besu[154406]: 2023-04-24 17:11:20.671-07:00 | vert.x-worker-thread-0 | INFO  | BackwardSyncContext | Starting a new backward sync session
Apr 24 17:11:21 hoenir besu[154406]: 2023-04-24 17:11:21.164-07:00 | nioEventLoopGroup-3-2 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2 completed, downloaded a total of 101 headers. Peers: 23
Apr 24 17:12:52 hoenir besu[154406]: 2023-04-24 17:12:52.830-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 3309 ms, time limit is 2000 ms
Apr 24 17:13:01 hoenir besu[154406]: 2023-04-24 17:13:01.153-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 2476 ms, time limit is 2000 ms
Apr 24 17:13:23 hoenir besu[154406]: 2023-04-24 17:13:23.947-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 2457 ms, time limit is 2000 ms
Apr 24 17:13:53 hoenir besu[154406]: 2023-04-24 17:13:53.876-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 2469 ms, time limit is 2000 ms
Apr 24 17:13:55 hoenir besu[154406]: 2023-04-24 17:13:55.519-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 4112 ms, time limit is 2000 ms
Apr 24 17:13:58 hoenir besu[154406]: 2023-04-24 17:13:57.153-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 5745 ms, time limit is 2000 ms
Apr 24 17:13:58 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:418)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.AbstractStreamingHasher.putBytes(AbstractStreamingHasher.java:96)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.AbstractHasher.putBytes(AbstractHasher.java:62)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.AbstractHasher.putBytes(AbstractHasher.java:29)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerTable.lambda$buildBloomFilter$d1c214ef$1(PeerTable.java:179)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.AbstractHasher.putObject(AbstractHasher.java:121)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.AbstractHashFunction.hashObject(AbstractHashFunction.java:36)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.BloomFilterStrategies$2.mightContain(BloomFilterStrategies.java:130)
Apr 24 17:13:58 hoenir besu[154406]:         at com.google.common.hash.BloomFilter.mightContain(BloomFilter.java:148)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerTable.get(PeerTable.java:86)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.lambda$getPeersFromNeighborsPacket$7(PeerDiscoveryController.java:420)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.getPeersFromNeighborsPacket(PeerDiscoveryController.java:421)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.lambda$onMessage$4(PeerDiscoveryController.java:348)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.Optional.ifPresent(Optional.java:178)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.onMessage(PeerDiscoveryController.java:345)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.PeerDiscoveryAgent.lambda$handleIncomingPacket$5(PeerDiscoveryAgent.java:307)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.util.Optional.ifPresent(Optional.java:178)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.PeerDiscoveryAgent.handleIncomingPacket(PeerDiscoveryAgent.java:307)
Apr 24 17:13:58 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.VertxPeerDiscoveryAgent.lambda$handlePacket$10(VertxPeerDiscoveryAgent.java:297)
Apr 24 17:13:58 hoenir besu[154406]:         at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
Apr 24 17:13:58 hoenir besu[154406]:         at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Apr 24 17:13:58 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:13:58 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:14:01 hoenir besu[154406]: 2023-04-24 17:13:58.784-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 7377 ms, time limit is 2000 ms
Apr 24 17:14:01 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.nio.ByteBuffer.wrap(ByteBuffer.java:418)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.AbstractStreamingHasher.putBytes(AbstractStreamingHasher.java:96)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.AbstractHasher.putBytes(AbstractHasher.java:62)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.AbstractHasher.putBytes(AbstractHasher.java:29)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerTable.lambda$buildBloomFilter$d1c214ef$1(PeerTable.java:179)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.AbstractHasher.putObject(AbstractHasher.java:121)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.AbstractHashFunction.hashObject(AbstractHashFunction.java:36)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.BloomFilterStrategies$2.mightContain(BloomFilterStrategies.java:130)
Apr 24 17:14:01 hoenir besu[154406]:         at com.google.common.hash.BloomFilter.mightContain(BloomFilter.java:148)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerTable.get(PeerTable.java:86)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.lambda$getPeersFromNeighborsPacket$7(PeerDiscoveryController.java:420)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.getPeersFromNeighborsPacket(PeerDiscoveryController.java:421)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.lambda$onMessage$4(PeerDiscoveryController.java:348)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.Optional.ifPresent(Optional.java:178)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.onMessage(PeerDiscoveryController.java:345)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.PeerDiscoveryAgent.lambda$handleIncomingPacket$5(PeerDiscoveryAgent.java:307)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.util.Optional.ifPresent(Optional.java:178)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.PeerDiscoveryAgent.handleIncomingPacket(PeerDiscoveryAgent.java:307)
Apr 24 17:14:01 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.VertxPeerDiscoveryAgent.lambda$handlePacket$10(VertxPeerDiscoveryAgent.java:297)
Apr 24 17:14:01 hoenir besu[154406]:         at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
Apr 24 17:14:01 hoenir besu[154406]:         at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Apr 24 17:14:01 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:14:01 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:14:06 hoenir besu[154406]: 2023-04-24 17:14:06.995-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 2475 ms, time limit is 2000 ms
Apr 24 17:14:08 hoenir besu[154406]: 2023-04-24 17:14:08.650-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 4131 ms, time limit is 2000 ms
Apr 24 17:14:10 hoenir besu[154406]: 2023-04-24 17:14:10.282-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 5762 ms, time limit is 2000 ms
Apr 24 17:14:10 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:14:10 hoenir besu[154406]:         at com.google.common.net.InetAddresses.textToNumericFormatV4(InetAddresses.java:226)
Apr 24 17:14:10 hoenir besu[154406]:         at com.google.common.net.InetAddresses.ipStringToBytes(InetAddresses.java:215)
Apr 24 17:14:10 hoenir besu[154406]:         at com.google.common.net.InetAddresses.forString(InetAddresses.java:148)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.Endpoint.encodeInline(Endpoint.java:145)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.Endpoint.encodeStandalone(Endpoint.java:133)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PingPacketData.writeTo(PingPacketData.java:139)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.Packet.encode(Packet.java:138)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.VertxPeerDiscoveryAgent.sendOutgoingPacket(VertxPeerDiscoveryAgent.java:171)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.PeerDiscoveryAgent.handleOutgoingPacket(PeerDiscoveryAgent.java:317)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.sendPacket(PeerDiscoveryController.java:605)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.internal.PeerDiscoveryController.lambda$bond$10(PeerDiscoveryController.java:546)
Apr 24 17:14:10 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718)
Apr 24 17:14:10 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
Apr 24 17:14:10 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
Apr 24 17:14:10 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.p2p.discovery.VertxPeerDiscoveryAgent$VertxAsyncExecutor.lambda$execute$1(VertxPeerDiscoveryAgent.java:327)
Apr 24 17:14:10 hoenir besu[154406]:         at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
Apr 24 17:14:10 hoenir besu[154406]:         at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Apr 24 17:14:10 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:14:10 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:14:25 hoenir besu[154406]: 2023-04-24 17:14:25.080-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 3289 ms, time limit is 2000 ms
Apr 24 17:14:30 hoenir besu[154406]: 2023-04-24 17:14:30.055-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 2482 ms, time limit is 2000 ms
Apr 24 17:14:33 hoenir besu[154406]: 2023-04-24 17:14:33.510-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 3452 ms, time limit is 2000 ms
Apr 24 17:14:57 hoenir besu[154406]: 2023-04-24 17:14:57.566-07:00 | nioEventLoopGroup-3-2 | WARN  | DefaultBlockchain | Chain Reorganization +0 new / -11125 old
Apr 24 17:14:57 hoenir besu[154406]:        Old - hash: 0xdf2c09de71884b49fc007c9bcd7293d371aa606613eaeffba094592453a78c07, height: 17119619
Apr 24 17:14:57 hoenir besu[154406]:        New - hash: 0x55cc01ae84b1b7189f56492146c4cb61102a27061e2dbc32c2460e8484240dc7, height: 17108494
Apr 24 17:14:57 hoenir besu[154406]:   Ancestor - hash: 0x55cc01ae84b1b7189f56492146c4cb61102a27061e2dbc32c2460e8484240dc7, height: 17108494
Apr 24 17:16:04 hoenir besu[154406]: 2023-04-24 17:16:04.292-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0xdf2c09de71884b49fc007c9bcd7293d371aa606613eaeffba094592453a78c07, finalized: 0x348d03191b0f28681e36ad35a862e9db97242debbac8aa71ae374d97b3caf8d4, safeBlockHash: 0x70c9c432ac81145083746f6e22082ccc92ae094bccf4e29d7e1ced79a5026881
Apr 24 17:16:05 hoenir besu[154406]: 2023-04-24 17:16:05.980-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,620 / 81 tx / 16 ws / base fee 36.11 gwei / 29,965,929 (99.9%) gas / (0xe3bc1e395a7b6b92ae6a572932f5245e2f9104242ae76e3aa418675489342a46) in 1.677s. Peers: 0
Apr 24 17:16:07 hoenir besu[154406]: 2023-04-24 17:16:07.360-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,621 / 96 tx / 16 ws / base fee 40.61 gwei / 13,459,677 (44.9%) gas / (0xa7d80dc8bfbdd2981464447ba40504228e17446e010d4aaa1985fbdd7d65e243) in 0.509s. Peers: 1
Apr 24 17:16:08 hoenir besu[154406]: 2023-04-24 17:16:08.902-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,622 / 121 tx / 16 ws / base fee 40.09 gwei / 10,523,656 (35.1%) gas / (0x42cc65abd4957f00e8cf2dd248dc286e335ce0ef640ddb1f01bb4f1d3b4eaf15) in 0.457s. Peers: 1
Apr 24 17:16:09 hoenir besu[154406]: 2023-04-24 17:16:09.582-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,623 / 102 tx / 16 ws / base fee 38.60 gwei / 8,904,490 (29.7%) gas / (0x0a251c63aa088ccf910ad73f25791539ba2dc35daac81e6a3e3bda7d1cf93e72) in 0.356s. Peers: 2
Apr 24 17:16:10 hoenir besu[154406]: 2023-04-24 17:16:10.255-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,624 / 126 tx / 16 ws / base fee 36.64 gwei / 10,503,896 (35.0%) gas / (0x6b0021daa4c1534bd204472bcbe5f8ce413f9931b9d7b612e319f1ea1ae2c667) in 0.376s. Peers: 2
Apr 24 17:16:11 hoenir besu[154406]: 2023-04-24 17:16:11.889-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,625 / 76 tx / 16 ws / base fee 35.26 gwei / 29,974,313 (99.9%) gas / (0xfc5e1868eb8eb03a272c1925b0f910eecdc6e8df90e81bd1a3e45a69e6a4a7ed) in 1.366s. Peers: 0
Apr 24 17:16:13 hoenir besu[154406]: 2023-04-24 17:16:13.139-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,626 / 104 tx / 16 ws / base fee 39.66 gwei / 9,496,552 (31.7%) gas / (0x6d04498951ee75f6eb1d5fccdad46f4f259f017da4ef050d169e036d964ed9bb) in 0.321s. Peers: 0
Apr 24 17:16:14 hoenir besu[154406]: 2023-04-24 17:16:14.169-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,627 / 109 tx / 16 ws / base fee 37.85 gwei / 10,258,835 (34.2%) gas / (0x2242e120094a89fedf4cf1c2f5f7700d5ecd9a1516dc2a836e235571f8a7402f) in 0.326s. Peers: 0
Apr 24 17:16:15 hoenir besu[154406]: 2023-04-24 17:16:15.136-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,628 / 113 tx / 16 ws / base fee 36.35 gwei / 7,542,566 (25.1%) gas / (0x4b68530e7bd74956117e72988fbcb4b4320ab439c4eeb0affebd68244e2dfda2) in 0.293s. Peers: 3
Apr 24 17:16:16 hoenir besu[154406]: 2023-04-24 17:16:16.892-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,629 / 78 tx / 16 ws / base fee 34.09 gwei / 29,983,728 (99.9%) gas / (0x39467d3e3a1d1a690a3309d121f4ef5bad2c6c5d94388605b5f54997848bb0b3) in 1.086s. Peers: 0
Apr 24 17:16:18 hoenir besu[154406]: 2023-04-24 17:16:18.161-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,630 / 121 tx / 16 ws / base fee 38.35 gwei / 13,619,144 (45.4%) gas / (0x66fa05a8dbae50d3f01bdcfd2b79b81c4882159f74cfb85262ca31a832fd3a9d) in 0.395s. Peers: 0
Apr 24 17:16:19 hoenir besu[154406]: 2023-04-24 17:16:19.098-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,631 / 84 tx / 16 ws / base fee 37.91 gwei / 6,502,656 (21.7%) gas / (0x70b8efc4d470cd5f6a41d16093484b298913e9e46adeb468a79f2ac5fae6bfc4) in 0.282s. Peers: 0
Apr 24 17:16:20 hoenir besu[154406]: 2023-04-24 17:16:20.186-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,632 / 159 tx / 16 ws / base fee 35.22 gwei / 13,799,072 (46.0%) gas / (0x76274e1b3bfe41c0b509c739e679835f0faafa7ac11ff84d8f44936a163d2f6a) in 0.516s. Peers: 0
Apr 24 17:16:21 hoenir besu[154406]: 2023-04-24 17:16:21.180-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,633 / 96 tx / 16 ws / base fee 34.87 gwei / 9,431,991 (31.4%) gas / (0x0d6fdf7fe90c6528c16c6b6ec2f3509eda2037aef3c3bf56ca730e1cc251946a) in 0.326s. Peers: 1
Apr 24 17:16:22 hoenir besu[154406]: 2023-04-24 17:16:22.300-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,634 / 145 tx / 16 ws / base fee 33.25 gwei / 13,687,281 (45.6%) gas / (0x2f5148cf12f428e1789ab25fef86204f019fb36427414fef6c56f46cf07e1a53) in 0.509s. Peers: 0
Apr 24 17:16:23 hoenir besu[154406]: 2023-04-24 17:16:23.412-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,635 / 132 tx / 16 ws / base fee 32.89 gwei / 13,023,004 (43.4%) gas / (0x510a13e62160d2da43b7e383a96c12fd65fdbc90711b13cf18ef80e659f388e5) in 0.477s. Peers: 0
Apr 24 17:16:25 hoenir besu[154406]: 2023-04-24 17:16:25.263-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,636 / 137 tx / 16 ws / base fee 32.35 gwei / 29,558,104 (98.5%) gas / (0x80a552f3320696c19d6f7e2515a1dcfe3d922e1ec5754b1d6d0cbe648ee0316f) in 1.202s. Peers: 0
Apr 24 17:16:26 hoenir besu[154406]: 2023-04-24 17:16:26.420-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,637 / 103 tx / 16 ws / base fee 36.27 gwei / 9,349,509 (31.2%) gas / (0x0969385e97da478960dfda5867543ee46fc9038acb5c1ea3587651302f338477) in 0.335s. Peers: 0
Apr 24 17:16:27 hoenir besu[154406]: 2023-04-24 17:16:27.469-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,638 / 106 tx / 16 ws / base fee 34.56 gwei / 12,923,880 (43.1%) gas / (0x05125a4c13a35fa2b541be3bac604d785fcb4100e4acf24ecfd7c804526dc171) in 0.425s. Peers: 0
Apr 24 17:16:28 hoenir besu[154406]: 2023-04-24 17:16:28.448-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,639 / 108 tx / 16 ws / base fee 33.96 gwei / 9,101,295 (30.3%) gas / (0x0e5b450eac2ba6115e5a381a22a34dcdb8cf374bbf49e7edcde9b8aaf7544ec2) in 0.339s. Peers: 0
Apr 24 17:16:30 hoenir besu[154406]: 2023-04-24 17:16:30.257-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,640 / 95 tx / 16 ws / base fee 32.29 gwei / 29,973,799 (99.9%) gas / (0x928a18d8bb1bd8257b81eb496f82e95c10a6ea68c54507ea5b75afcb905b04ba) in 1.255s. Peers: 0
Apr 24 17:16:32 hoenir besu[154406]: 2023-04-24 17:16:32.221-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,641 / 90 tx / 16 ws / base fee 36.32 gwei / 10,587,053 (35.3%) gas / (0x109a70934246809819744ea9b6f8fe3ed7355f6797faa0de1240d8d88836ea6c) in 0.463s. Peers: 0
Apr 24 17:16:33 hoenir besu[154406]: 2023-04-24 17:16:33.158-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,642 / 97 tx / 16 ws / base fee 34.99 gwei / 14,491,807 (48.3%) gas / (0x592f3c2a7493973af2842e709c93116d77f4052903962932bdc1a985967db2cd) in 0.365s. Peers: 0
Apr 24 17:16:34 hoenir besu[154406]: 2023-04-24 17:16:34.895-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,643 / 120 tx / 16 ws / base fee 34.84 gwei / 29,889,807 (99.6%) gas / (0x78333bf59fc46a3dd1adf6db525d5793105f2b81445a3beffabda2567ca4030c) in 1.243s. Peers: 0
Apr 24 17:16:36 hoenir besu[154406]: 2023-04-24 17:16:36.024-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,644 / 78 tx / 16 ws / base fee 39.16 gwei / 9,606,663 (32.0%) gas / (0x5b7e4fb6b8bea9e4e08515e794b70088053b71f28c78513f24934a8c28ce1bd7) in 0.323s. Peers: 0
Apr 24 17:16:36 hoenir besu[154406]: 2023-04-24 17:16:36.838-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,645 / 87 tx / 16 ws / base fee 37.40 gwei / 7,462,085 (24.9%) gas / (0x2c3d4b3faf32ef94801c9ee35fbef7df0f11517385a9367b9f7b0e0c558a173d) in 0.278s. Peers: 17
Apr 24 17:16:37 hoenir besu[154406]: 2023-04-24 17:16:37.697-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,646 / 99 tx / 16 ws / base fee 35.05 gwei / 9,328,296 (31.1%) gas / (0xd02ac90a9da77088ddab8c66f1f3c01acb57964fce3e5f5a11ed49ad3be05f39) in 0.349s. Peers: 2
Apr 24 17:16:38 hoenir besu[154406]: 2023-04-24 17:16:38.650-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,647 / 154 tx / 16 ws / base fee 33.40 gwei / 15,700,564 (52.3%) gas / (0xb2b7b9944fe0ff37fe56c0dc3901796353884b784852b2d78c1191f19141926a) in 0.479s. Peers: 2
Apr 24 17:16:39 hoenir besu[154406]: 2023-04-24 17:16:39.338-07:00 | nioEventLoopGroup-3-4 | INFO  | BackwardSyncStep | Backward sync phase 1 of 2, 6.25% completed, downloaded 2 headers of at least 32. Peers: 3
Apr 24 17:16:39 hoenir besu[154406]: 2023-04-24 17:16:39.544-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,648 / 91 tx / 16 ws / base fee 33.59 gwei / 9,033,247 (30.1%) gas / (0x370acbd61626287394e6d75ac613ddd494454f6ce8e22396c2bce8bb43de4282) in 0.349s. Peers: 2
Apr 24 17:16:41 hoenir besu[154406]: 2023-04-24 17:16:41.271-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,649 / 169 tx / 16 ws / base fee 31.92 gwei / 29,979,317 (99.9%) gas / (0x4353a23d1d2b27c55d934383fd9ea71c85cc33eca2a0d4fe98c3228a38d118b5) in 1.260s. Peers: 1
Apr 24 17:16:42 hoenir besu[154406]: 2023-04-24 17:16:42.511-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,650 / 100 tx / 16 ws / base fee 35.91 gwei / 17,424,802 (58.1%) gas / (0x826817355f33aa950520bf225bb10c7dac8170c687922f599dfe140cadaf05f4) in 0.597s. Peers: 2
Apr 24 17:16:44 hoenir besu[154406]: 2023-04-24 17:16:44.026-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,651 / 108 tx / 16 ws / base fee 36.63 gwei / 7,633,545 (25.4%) gas / (0xd603533fcf4a36521d9c3e90b1ce7f8037220f292f8d413766f5272dc8a73e91) in 0.293s. Peers: 1
Apr 24 17:16:44 hoenir besu[154406]: 2023-04-24 17:16:44.988-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,652 / 121 tx / 16 ws / base fee 34.38 gwei / 11,136,422 (37.1%) gas / (0xc8d369fda2e8870efe30d65d73d77541f65c7762af0b8e31f3787e60f7c3ecd2) in 0.489s. Peers: 0
Apr 24 17:16:45 hoenir besu[154406]: 2023-04-24 17:16:45.974-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,653 / 112 tx / 16 ws / base fee 33.28 gwei / 10,966,064 (36.6%) gas / (0x4af60f89360bd73a1d51998ef7d7b2bc26a98c8c736adfcefceaaa48b3cd86a4) in 0.544s. Peers: 0
Apr 24 17:16:48 hoenir besu[154406]: 2023-04-24 17:16:48.303-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,654 / 125 tx / 16 ws / base fee 32.16 gwei / 29,699,922 (99.0%) gas / (0x53a23b5dd92fe56d89c33e289297e75ac9ef74603320ec4a32c2eb40a8a1433f) in 1.435s. Peers: 0
Apr 24 17:17:09 hoenir besu[154406]: 2023-04-24 17:17:09.037-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,655 / 175 tx / 16 ws / base fee 36.10 gwei / 16,922,440 (56.4%) gas / (0x5ff1cc1ad1438d90f6789d15584b7da4140ffefb32a64d49fa5534c3f78e4b49) in 0.657s. Peers: 0
Apr 24 17:17:09 hoenir besu[154406]: 2023-04-24 17:17:09.360-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineForkchoiceUpdated | VALID for fork-choice-update: head: 0x5ff1cc1ad1438d90f6789d15584b7da4140ffefb32a64d49fa5534c3f78e4b49, finalized: 0x70c9c432ac81145083746f6e22082ccc92ae094bccf4e29d7e1ced79a5026881, safeBlockHash: 0xa7d80dc8bfbdd2981464447ba40504228e17446e010d4aaa1985fbdd7d65e243
Apr 24 17:17:09 hoenir besu[154406]: 2023-04-24 17:17:09.746-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,656 / 110 tx / 16 ws / base fee 36.68 gwei / 10,947,109 (36.5%) gas / (0xd577db27e3a23c9aba9c6f20d5331dc0403133815584627c1c49e15869b8c9ae) in 0.359s. Peers: 0
Apr 24 17:17:12 hoenir besu[154406]: 2023-04-24 17:17:12.602-07:00 | vert.x-worker-thread-0 | INFO  | AbstractEngineNewPayload | Imported #17,119,657 / 105 tx / 16 ws / base fee 35.44 gwei / 9,261,830 (30.9%) gas / (0xaca390eb270ba4363f48a57552caecdfd0f1e839951046183c8a1130693a157d) in 0.450s. Peers: 0
Apr 24 17:18:17 hoenir besu[154406]: 2023-04-24 17:18:17.146-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 3259 ms, time limit is 2000 ms
Apr 24 17:18:25 hoenir besu[154406]: 2023-04-24 17:18:25.274-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 60786 ms, time limit is 60000 ms
Apr 24 17:18:25 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:25 hoenir besu[154406]:         at org.apache.tuweni.bytes.Bytes.compareTo(Bytes.java:1701)
Apr 24 17:18:25 hoenir besu[154406]:         at org.apache.tuweni.bytes.Bytes.compareTo(Bytes.java:46)
Apr 24 17:18:25 hoenir besu[154406]:         at java.base/java.util.TreeMap.getEntry(TreeMap.java:350)
Apr 24 17:18:25 hoenir besu[154406]:         at java.base/java.util.TreeMap.get(TreeMap.java:279)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:221)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.operation.SLoadOperation.execute(SLoadOperation.java:64)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:307)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:511)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:420)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:25 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:25 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:25 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:25 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:25 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:25 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:25 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:25 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:25 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:26 hoenir besu[154406]: 2023-04-24 17:18:25.273-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 60786 ms, time limit is 60000 ms
Apr 24 17:18:26 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:26 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:26 hoenir besu[154406]: 2023-04-24 17:18:26.915-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 62428 ms, time limit is 60000 ms
Apr 24 17:18:26 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:26 hoenir besu[154406]:         at org.rocksdb.Transaction.get(Native Method)
Apr 24 17:18:26 hoenir besu[154406]:         at org.rocksdb.Transaction.get(Transaction.java:278)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBSnapshotTransaction.get(RocksDBSnapshotTransaction.java:102)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueSnapshot.get(RocksDBColumnarKeyValueSnapshot.java:69)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getAccountStorageTrieNode(BonsaiWorldStateKeyValueStorage.java:140)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccountStorageTrieNode(BonsaiSnapshotWorldStateKeyValueStorage.java:91)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.lambda$getStorageValueBySlotHash$5(BonsaiWorldStateKeyValueStorage.java:190)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNodeFactory.retrieve(StoredNodeFactory.java:96)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:132)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:65)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:119)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getStorageValueBySlotHash(BonsaiWorldStateKeyValueStorage.java:194)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getStorageValueBySlotHash(BonsaiSnapshotWorldStateKeyValueStorage.java:128)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.getStorageValueBySlotHash(BonsaiWorldState.java:515)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getStorageValueBySlotHash(BonsaiWorldStateUpdateAccumulator.java:410)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getStorageValue(BonsaiWorldStateUpdateAccumulator.java:390)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.getStorageValue(BonsaiAccount.java:211)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.UpdateTrackingAccount.getStorageValue(UpdateTrackingAccount.java:231)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.operation.SLoadOperation.execute(SLoadOperation.java:64)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:307)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:511)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:420)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:26 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:26 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:26 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:26 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:27 hoenir besu[154406]: 2023-04-24 17:18:26.915-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 62428 ms, time limit is 60000 ms
Apr 24 17:18:27 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:27 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:27 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:27 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:27 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:27 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:27 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:27 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:27 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:27 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:27 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:27 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:28 hoenir besu[154406]: 2023-04-24 17:18:28.547-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 64060 ms, time limit is 60000 ms
Apr 24 17:18:28 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:28 hoenir besu[154406]:         at org.apache.tuweni.bytes.Bytes.toArray(Bytes.java:1562)
Apr 24 17:18:28 hoenir besu[154406]:         at org.apache.tuweni.bytes.Bytes.toArray(Bytes.java:1552)
Apr 24 17:18:28 hoenir besu[154406]:         at org.apache.tuweni.bytes.Bytes.toArrayUnsafe(Bytes.java:1588)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.evm.operation.AddOperation.staticOperation(AddOperation.java:53)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:160)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:511)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:420)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:28 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:28 hoenir besu[154406]: 2023-04-24 17:18:28.547-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 64060 ms, time limit is 60000 ms
Apr 24 17:18:28 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:28 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:28 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:28 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:28 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:28 hoenir besu[154406]: 2023-04-24 17:18:28.547-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 2444 ms, time limit is 2000 ms
Apr 24 17:18:30 hoenir besu[154406]: 2023-04-24 17:18:30.175-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 65688 ms, time limit is 60000 ms
Apr 24 17:18:30 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:30 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:30 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:30 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:30 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:30 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:30 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:30 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:30 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:30 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:30 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:30 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:31 hoenir besu[154406]: 2023-04-24 17:18:30.175-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 65688 ms, time limit is 60000 ms
Apr 24 17:18:31 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.evm.operation.AbstractFixedCostOperation.execute(AbstractFixedCostOperation.java:71)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:307)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:511)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:420)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:31 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:31 hoenir besu[154406]: 2023-04-24 17:18:31.798-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 67311 ms, time limit is 60000 ms
Apr 24 17:18:31 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:31 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:31 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:31 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:31 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:33 hoenir besu[154406]: 2023-04-24 17:18:33.410-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 68923 ms, time limit is 60000 ms
Apr 24 17:18:33 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:33 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:33 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:33 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:33 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:33 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:33 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:33 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:33 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:33 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:33 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:33 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:39 hoenir besu[154406]: 2023-04-24 17:18:35.835-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 70541 ms, time limit is 60000 ms
Apr 24 17:18:39 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:39 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:39 hoenir besu[154406]: 2023-04-24 17:18:39.110-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 74623 ms, time limit is 60000 ms
Apr 24 17:18:39 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:39 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:39 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:39 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:39 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:40 hoenir besu[154406]: 2023-04-24 17:18:40.751-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 76264 ms, time limit is 60000 ms
Apr 24 17:18:40 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetProtocolSpecs.berlinTransactionReceiptFactory(MainnetProtocolSpecs.java:893)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:152)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:40 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:40 hoenir besu[154406]: 2023-04-24 17:18:40.751-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 76264 ms, time limit is 60000 ms
Apr 24 17:18:40 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:40 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:40 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:40 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:40 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:42 hoenir besu[154406]: 2023-04-24 17:18:42.374-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 77887 ms, time limit is 60000 ms
Apr 24 17:18:42 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:42 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:42 hoenir besu[154406]: 2023-04-24 17:18:42.374-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 77887 ms, time limit is 60000 ms
Apr 24 17:18:42 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:320)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:511)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:420)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:42 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:42 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:42 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:42 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:44 hoenir besu[154406]: 2023-04-24 17:18:44.004-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 79516 ms, time limit is 60000 ms
Apr 24 17:18:44 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.evm.operation.MulOperation.staticOperation(MulOperation.java:56)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:163)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:174)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:192)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:511)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:420)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:44 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:44 hoenir besu[154406]: 2023-04-24 17:18:44.005-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 79517 ms, time limit is 60000 ms
Apr 24 17:18:44 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:44 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:44 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:44 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:44 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:45 hoenir besu[154406]: 2023-04-24 17:18:45.620-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 81133 ms, time limit is 60000 ms
Apr 24 17:18:45 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:45 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:45 hoenir besu[154406]: 2023-04-24 17:18:45.622-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 81133 ms, time limit is 60000 ms
Apr 24 17:18:45 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:45 hoenir besu[154406]:         at org.rocksdb.Transaction.get(Native Method)
Apr 24 17:18:45 hoenir besu[154406]:         at org.rocksdb.Transaction.get(Transaction.java:278)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBSnapshotTransaction.get(RocksDBSnapshotTransaction.java:102)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueSnapshot.get(RocksDBColumnarKeyValueSnapshot.java:69)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getAccountStateTrieNode(BonsaiWorldStateKeyValueStorage.java:127)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccountStateTrieNode(BonsaiSnapshotWorldStateKeyValueStorage.java:83)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNodeFactory.retrieve(StoredNodeFactory.java:96)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:132)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:65)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:119)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getAccount(BonsaiWorldStateKeyValueStorage.java:109)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccount(BonsaiSnapshotWorldStateKeyValueStorage.java:73)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:470)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.loadAccount(BonsaiWorldStateUpdateAccumulator.java:179)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getForMutation(BonsaiWorldStateUpdateAccumulator.java:164)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getForMutation(BonsaiWorldStateUpdateAccumulator.java:55)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:113)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.get(BonsaiWorldStateUpdateAccumulator.java:107)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:401)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:45 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:45 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:45 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:45 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:45 hoenir besu[154406]: 2023-04-24 17:18:45.622-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#34,vert.x-eventloop-thread-1,5,main] has been blocked for 3243 ms, time limit is 2000 ms
Apr 24 17:18:47 hoenir besu[154406]: 2023-04-24 17:18:47.239-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 82752 ms, time limit is 60000 ms
Apr 24 17:18:47 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:47 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:47 hoenir besu[154406]: 2023-04-24 17:18:47.239-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 82752 ms, time limit is 60000 ms
Apr 24 17:18:47 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:47 hoenir besu[154406]:         at org.rocksdb.Transaction.get(Native Method)
Apr 24 17:18:47 hoenir besu[154406]:         at org.rocksdb.Transaction.get(Transaction.java:278)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBSnapshotTransaction.get(RocksDBSnapshotTransaction.java:102)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.plugin.services.storage.rocksdb.segmented.RocksDBColumnarKeyValueSnapshot.get(RocksDBColumnarKeyValueSnapshot.java:69)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getAccountStateTrieNode(BonsaiWorldStateKeyValueStorage.java:127)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccountStateTrieNode(BonsaiSnapshotWorldStateKeyValueStorage.java:83)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNodeFactory.retrieve(StoredNodeFactory.java:96)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:132)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:65)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:80)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:66)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:119)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getAccount(BonsaiWorldStateKeyValueStorage.java:109)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccount(BonsaiSnapshotWorldStateKeyValueStorage.java:73)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:470)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.loadAccount(BonsaiWorldStateUpdateAccumulator.java:179)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getForMutation(BonsaiWorldStateUpdateAccumulator.java:164)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getForMutation(BonsaiWorldStateUpdateAccumulator.java:55)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:113)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.get(BonsaiWorldStateUpdateAccumulator.java:107)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:401)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:47 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:47 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:47 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:47 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:48 hoenir besu[154406]: 2023-04-24 17:18:48.895-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 84408 ms, time limit is 60000 ms
Apr 24 17:18:48 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinTask.awaitDone(ForkJoinTask.java:461)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:668)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:765)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.commit(BonsaiWorldStateUpdateAccumulator.java:279)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:148)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:48 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:48 hoenir besu[154406]: 2023-04-24 17:18:48.895-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#53,vert.x-worker-thread-7,5,main] has been blocked for 84408 ms, time limit is 60000 ms
Apr 24 17:18:48 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/jdk.internal.misc.Unsafe.park(Native Method)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3745)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3690)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:89)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:42)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.AuthenticatedJsonRpcProcessor.process(AuthenticatedJsonRpcProcessor.java:51)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
Apr 24 17:18:48 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:85)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:48 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:48 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:48 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:51 hoenir besu[154406]: 2023-04-24 17:18:50.537-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 86049 ms, time limit is 60000 ms
Apr 24 17:18:51 hoenir besu[154406]: io.vertx.core.VertxException: Thread blocked
Apr 24 17:18:51 hoenir besu[154406]:         at org.apache.tuweni.bytes.AbstractBytes.computeHashcode(AbstractBytes.java:60)
Apr 24 17:18:51 hoenir besu[154406]:         at org.apache.tuweni.bytes.AbstractBytes.hashCode(AbstractBytes.java:69)
Apr 24 17:18:51 hoenir besu[154406]:         at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:72)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.services.kvstore.LayeredKeyValueStorage.get(LayeredKeyValueStorage.java:74)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiWorldStateKeyValueStorage.getAccount(BonsaiWorldStateKeyValueStorage.java:99)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccount(BonsaiSnapshotWorldStateKeyValueStorage.java:73)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:470)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.loadAccount(BonsaiWorldStateUpdateAccumulator.java:179)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getForMutation(BonsaiWorldStateUpdateAccumulator.java:164)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getForMutation(BonsaiWorldStateUpdateAccumulator.java:55)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:113)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.getAccount(BonsaiWorldStateUpdateAccumulator.java:118)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.WorldUpdater.getOrCreate(WorldUpdater.java:77)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.evm.worldstate.WorldUpdater.getOrCreateSenderAccount(WorldUpdater.java:89)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:289)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:123)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:78)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:228)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:137)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:85)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.validateBlock(MergeCoordinator.java:498)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.MergeCoordinator.rememberBlock(MergeCoordinator.java:528)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.consensus.merge.blockcreation.TransitionCoordinator.rememberBlock(TransitionCoordinator.java:157)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine.AbstractEngineNewPayload.syncResponse(AbstractEngineNewPayload.java:239)
Apr 24 17:18:51 hoenir besu[154406]:         at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.lambda$response$0(ExecutionEngineJsonRpcMethod.java:73)
Apr 24 17:18:51 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:159)
Apr 24 17:18:51 hoenir besu[154406]:         at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
Apr 24 17:18:51 hoenir besu[154406]:         at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:157)
Apr 24 17:18:51 hoenir besu[154406]:         at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
Apr 24 17:18:51 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Apr 24 17:18:51 hoenir besu[154406]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Apr 24 17:18:51 hoenir besu[154406]:         at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Apr 24 17:18:51 hoenir besu[154406]:         at java.base/java.lang.Thread.run(Thread.java:1589)
Apr 24 17:18:52 hoenir besu[154406]: 2023-04-24 17:18:52.159-07:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[#79,vert.x-worker-thread-0,5,main] has been blocked for 87672 ms, time limit is 60000 ms
non-fungible-nelson commented 1 year ago

prioritizing looking at this and reevaluating the timeouts.

non-fungible-nelson commented 1 year ago

Another example

non-fungible-nelson commented 1 year ago

as a mitigation in the meantime - try resyncing your CL optimistically (checkpoint sync). A lot of these are caused by CLs trying to get back in sync with Besu and spamming a ton of Engine API calls.

non-fungible-nelson commented 1 year ago

Closing as not able to reproduce on latest (23.4.4). We have made significant improvements to bonsai safety.

lakshay-zeeve commented 8 months ago

Facing same issue with 23.7.2 image