search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.26k stars 3.59k forks source link

bookkeeper is running out of direct memory #19334

Open sindhushreem opened 1 year ago

sindhushreem commented 1 year ago

Search before asking

Version

2.8.2

Minimal reproduce step

bookeeper runnning for some days and ran out of direct memory .

What did you expect to see?

Bookkeeper should be running without any error . we installed 2.8.2 and bookkeeper is not up and running . We shall attach the heap dump form the bookie for the same .

What did you see instead?

INFO org.apache.bookkeeper.bookie.JournalChannel - Opening journal data/bookkeeper/journal/current/181400c6508.txn 2022-11-14T22:01:22,123 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Failed to start Component: bookie-server java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:695) ~[?:1.8.0_332] at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) ~[?:1.8.0_332] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[?:1.8.0_332] at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:247) ~[?:1.8.0_332] at sun.nio.ch.IOUtil.read(IOUtil.java:195) ~[?:1.8.0_332] at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159) ~[?:1.8.0_332] at org.apache.bookkeeper.bookie.JournalChannel.read(JournalChannel.java:261) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.bookie.Journal.fullRead(Journal.java:1194) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:833) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:995) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:961) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1015) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3] at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3] at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:406) [com.google.guava-guava-30.1-jre.jar:?] at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144) [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3] at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85) [org.apache.bookkeeper-bookkeeper-common-4.14.3.jar:4.14.3] at org.apache.bookkeeper.server.Main.doMain(Main.java:234) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.3.jar:4.14.3] 2022-11-14T22:01:22,129 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Calling uncaughtExceptionHandler 2022-11-14T22:01:22,129 [main] ERROR org.apache.bookkeeper.common.component.ComponentStarter - Triggered exceptionHandler of Component: bookie-server because of Exception in Thread: Thread[main,5,main] java.lang.OutOfMemoryError: Direct buffer memory

Anything else?

No response

Are you willing to submit a PR?

hangc0276 commented 1 year ago

Hi @sindhushreem, thank you for raising this issue. Could you share your direct memory configuration and the heap dump if possible?

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.

sindhushreem commented 1 year ago

Hi @sindhushreem, thank you for raising this issue. Could you share your direct memory configuration and the heap dump if possible?

Hi Hang , I have shared the direct memory configuration and heap dump with @codelipenghui

github-actions[bot] commented 1 year ago

The issue had no activity for 30 days, mark with Stale label.