java.lang.OutOfMemoryError: Java heap space

[binny1024]

Description

This happens when I synchronize blocks with blockscout

Acceptance Criteria

• [Criteria 1]

Steps to Reproduce (Bug)

1. [Step 1]
2. [Step 2]
3. [Step ...]

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Frequency: [What percentage of the time does it occur?] Not sure, ranging from ten minutes to half an hour

Logs (if a bug)

Please post relevant logs from Besu (and the consensus client, if running proof of stake) from before and after the issue.

0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:16.380+08:00 | nioEventLoopGroup-3-5 | INFO | BlockPropagationManager | Saved announced block for future import 20567797 (0x4eaa3490923e2e5d886cf37ca11c917eaef2eea9d208d301e92f54279726e422) - 22 saved block(s) 0|node4 | 2024-04-07 11:09:16.381+08:00 | nioEventLoopGroup-3-5 | INFO | BlockPropagationManager | Saved announced block for future import 20567798 (0x1a3e6a8513c47ac949c640874106bd3cc3751082f42c925e6c97ae9741b0e8e9) - 23 saved block(s) 0|node4 | 2024-04-07 11:09:16.384+08:00 | nioEventLoopGroup-3-5 | INFO | BlockPropagationManager | Saved announced block for future import 20567799 (0xb1fe88eb3c2f743c0c22cecc1d29668670ad2e7a24c3ec3f8ed66e984938c82d) - 24 saved block(s) 0|node4 | 2024-04-07 11:09:16.518+08:00 | nioEventLoopGroup-3-5 | INFO | BlockPropagationManager | Saved announced block for future import 20567800 (0x285932eefc9fed73ee5a6dd3a5a07728878eba5acf8694d4589df78fc93dbcd2) - 25 saved block(s) 0|node4 | 2024-04-07 11:09:17.103+08:00 | nioEventLoopGroup-3-5 | INFO | BlockPropagationManager | Saved announced block for future import 20567801 (0x44d56954c4c63517190e11abf0be0473b9e5b3cf61515cb7217a951b84c8cc4f) - 26 saved block(s) 0|node4 | 2024-04-07 11:09:22.199+08:00 | nioEventLoopGroup-3-2 | INFO | BlockPropagationManager | Saved announced block for future import 20567802 (0x510d97690f534791b870770a552b73517ad58344b35a32c41f69ad2da0ad2e83) - 27 saved block(s) 0|node4 | 2024-04-07 11:09:22.199+08:00 | vert.x-eventloop-thread-3 | ERROR | RoutingContext | Unhandled exception in router 0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:22.401+08:00 | nioEventLoopGroup-3-2 | INFO | BlockPropagationManager | Saved announced block for future import 20567803 (0x00d32f96c6cdbf50e59c8798ea5809ff69c7962fdee5a465a49c753081d99c61) - 28 saved block(s) 0|node4 | 2024-04-07 11:09:23.249+08:00 | nioEventLoopGroup-3-5 | INFO | BlockPropagationManager | Saved announced block for future import 20567804 (0x9da519e431994e490876cd65337f61b18504e65c4ed1169f9ae22d29176ff8fd) - 29 saved block(s) 0|node4 | 2024-04-07 11:09:40.765+08:00 | BftProcessorExecutor-IBFT-0 | ERROR | BftProcessor | BFT Mining thread has suffered a fatal error, mining has been halted 0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:41.952+08:00 | BftProcessorExecutor-IBFT-0 | INFO | BftProcessor | Shutting down BFT event processor 0|node4 | 2024-04-07 11:09:41.952+08:00 | nioEventLoopGroup-3-5 | ERROR | DeFramer | Exception while processing incoming message 0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:41.952+08:00 | vert.x-eventloop-thread-3 | ERROR | RoutingContext | Unhandled exception in router 0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:53.081+08:00 | nioEventLoopGroup-3-2 | ERROR | DeFramer | Exception while processing incoming message 0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:53.082+08:00 | EthScheduler-Workers-0 | INFO | MainnetBlockValidator | Optional[Parent block with hash 0x949b86a6a8dda0d4340cce22770b55a963ea9230098ac4127260ef1354e06574 not present]. Block 20567776 (0x9afaf2b1d4159b600da2ff0bbd2c983ad56fd4937b7e6e06acd10101fdffa6bd) 0|node4 | 2024-04-07 11:09:53.082+08:00 | vert.x-eventloop-thread-3 | ERROR | RoutingContext | Unhandled exception in router 0|node4 | java.lang.OutOfMemoryError: Java heap space 0|node4 | 2024-04-07 11:09:53.082+08:00 | EthScheduler-Workers-0 | ERROR | BlockPropagationManager | Error importing pending blocks 0|node4 | org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block at #20567776 (0x9afaf2b1d4159b600da2ff0bbd2c983ad56fd4937b7e6e06acd10101fdffa6bd) 0|node4 | at org.hyperledger.besu.ethereum.eth.sync.tasks.PersistBlockTask.executeTask(PersistBlockTask.java:206) 0|node4 | at org.hyperledger.besu.ethereum.eth.manager.task.AbstractEthTask.executeTaskTimed(AbstractEthTask.java:150) 0|node4 | at org.hyperledger.besu.ethereum.eth.manager.task.AbstractEthTask.run(AbstractEthTask.java:75) 0|node4 | at org.hyperledger.besu.ethereum.eth.sync.tasks.PersistBlockTask.lambda$forUnorderedBlocks$8(PersistBlockTask.java:165) 0|node4 | at org.hyperledger.besu.util.FutureUtils.propagateResult(FutureUtils.java:97) 0|node4 | at org.hyperledger.besu.ethereum.eth.manager.EthScheduler.lambda$scheduleSyncWorkerTask$0(EthScheduler.java:110) 0|node4 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) 0|node4 | at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 0|node4 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 0|node4 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 0|node4 | at java.base/java.lang.Thread.run(Thread.java:833) 0|node4 | 2024-04-07 11:09:53.086+08:00 | EthScheduler-Workers-0 | WARN | RetryingGetBlockFromPeersTask | Failed to get block 20567775 (0x949b86a6a8dda0d4340cce22770b55a963ea9230098ac4127260ef1354e06574) after 1 retries ....

tLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@18a01292 0|node4 | 2024-04-07 11:11:17.003+08:00 | nioEventLoopGroup-3-6 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@317e0816 0|node4 | 2024-04-07 11:11:17.004+08:00 | nioEventLoopGroup-3-4 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@38392966 0|node4 | 2024-04-07 11:11:17.004+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@7d858360 0|node4 | 2024-04-07 11:11:17.006+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@59b5d62c 0|node4 | 2024-04-07 11:11:17.006+08:00 | nioEventLoopGroup-3-4 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@3a31e217 0|node4 | 2024-04-07 11:11:17.007+08:00 | nioEventLoopGroup-3-6 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@755b5d56 0|node4 | 2024-04-07 11:11:17.007+08:00 | nioEventLoopGroup-3-6 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@75cf9421 0|node4 | 2024-04-07 11:11:17.007+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@3071caea 0|node4 | 2024-04-07 11:11:17.008+08:00 | nioEventLoopGroup-3-4 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@ce54a09 0|node4 | 2024-04-07 11:11:17.015+08:00 | nioEventLoopGroup-3-5 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@2e4c0102 0|node4 | 2024-04-07 11:11:17.016+08:00 | nioEventLoopGroup-3-5 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@5ec53a12 0|node4 | 2024-04-07 11:11:17.017+08:00 | nioEventLoopGroup-3-4 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@9c70773 0|node4 | 2024-04-07 11:11:17.017+08:00 | nioEventLoopGroup-3-5 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@8bf32b7 0|node4 | 2024-04-07 11:11:17.018+08:00 | nioEventLoopGroup-3-6 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@9f57695 0|node4 | 2024-04-07 11:11:17.018+08:00 | nioEventLoopGroup-3-4 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@78c46d34 0|node4 | 2024-04-07 11:11:17.018+08:00 | nioEventLoopGroup-3-6 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@488ae105 0|node4 | 2024-04-07 11:11:17.018+08:00 | nioEventLoopGroup-3-5 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@18f88732 0|node4 | 2024-04-07 11:11:17.019+08:00 | nioEventLoopGroup-3-5 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@56f2d13e 0|node4 | 2024-04-07 11:11:17.019+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@28dd6af4 0|node4 | 2024-04-07 11:11:17.019+08:00 | nioEventLoopGroup-3-6 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@6714a4af 0|node4 | 2024-04-07 11:11:17.019+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@7912720c 0|node4 | 2024-04-07 11:11:17.020+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@ee1cfb4 0|node4 | 2024-04-07 11:11:17.020+08:00 | nioEventLoopGroup-3-4 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@4dd01961 0|node4 | 2024-04-07 11:11:17.020+08:00 | nioEventLoopGroup-3-7 | WARN | BftEventQueue | Queue size exceeded trying to add new bft event org.hyperledger.besu.consensus.common.bft.events.BftReceivedMessageEvent@30f96218 0|nod

Versions (Add all that apply)

• Software version: [besu --version]

  besu/v22.10.0/linux-x86_64/oracle_openjdk-java-17
  2024-04-07 11:11:39.975+08:00 | main | INFO  | Besu | Using jemalloc

• Java version: [java -version]

openjdk version "17.0.1" 2021-10-19
OpenJDK Runtime Environment (build 17.0.1+12-39)
OpenJDK 64-Bit Server VM (build 17.0.1+12-39, mixed mode, sharing

• OS Name & Version: [cat /etc/*release]

  openjdk version "17.0.1" 2021-10-19
  OpenJDK Runtime Environment (build 17.0.1+12-39)
  OpenJDK 64-Bit Server VM (build 17.0.1+12-39, mixed mode, sharing)
  root@iZ0jle0x55k1uyo31vpqfzZ:~# 
  root@iZ0jle0x55k1uyo31vpqfzZ:~# 
  root@iZ0jle0x55k1uyo31vpqfzZ:~# 
  root@iZ0jle0x55k1uyo31vpqfzZ:~# 
  root@iZ0jle0x55k1uyo31vpqfzZ:~# cat /etc/*release
  DISTRIB_ID=Ubuntu
  DISTRIB_RELEASE=22.04
  DISTRIB_CODENAME=jammy
  DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS"
  PRETTY_NAME="Ubuntu 22.04.4 LTS"
  NAME="Ubuntu"
  VERSION_ID="22.04"
  VERSION="22.04.4 LTS (Jammy Jellyfish)"
  VERSION_CODENAME=jammy
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  UBUNTU_CODENAME=jammy

• Kernel Version: [uname -a]

  Linux iZ0jle0x55k1uyo31vpqfzZ 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

• Virtual Machine software & version: [vmware -v]
• Docker Version: [docker version]
• Cloud VM, type, size: [Amazon Web Services I3-large]
• Consensus Client & Version if using Proof of Stake: [e.g. Teku, Lighthouse, Prysm, Nimbus, Lodestar]

Smart contract information (If you're reporting an issue arising from deploying or calling a smart contract, please supply related information)

• Solidity version [solc --version]
• Repo with minimal set of deployable/reproducible contract code - please provide a link
• Please include specifics on how you are deploying/calling the contract
• Have you reproduced the issue on other eth clients

Additional Information (Add any of the following or anything else that may be relevant)

• Besu setup info - genesis file, config options
• System info - memory, CPU

[non-fungible-nelson]

Hello there! Here are some ideas --> you can manually increase the Java heap size. It seems you have more memory available on the system that can be allocated to Besu. Check out this page on how to do that.

You may also want to update your version of Besu. We have made substantial performance improvements to Besu since version 22.10.0. Thanks!

[ahamlat]

Agreed with @non-fungible-nelson, please consider upgrading to a newer Besu release. For instance, this PR https://github.com/hyperledger/besu/pull/5131 improves a lot memory usage on trace_replayBlockTransactions, and I believe blockscout uses this PRC method to display internal transactions. If you still see the same error with a Besu 24.3.3, please share your Besu configuration especially -Xmx and how to reproduce.