search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.42k stars 1.27k forks source link

consumer out of memory #3874

Closed sunny19930321 closed 5 years ago

sunny19930321 commented 5 years ago

When the consumer is consuming, when the state is transitioning, memory out of memory, consumption stops. What's going on here server configuration（6+32G, jvm16G） java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.helix.messaging.handling.HelixStateTransitionHandler.invoke(HelixStateTransitionHandler.java:430) at org.apache.helix.messaging.handling.HelixStateTransitionHandler.handleMessage(HelixStateTransitionHandler.java:344) at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:96) at org.apache.helix.messaging.handling.HelixTask.call(HelixTask.java:47) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:693) at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) at com.linkedin.pinot.core.segment.memory.PinotByteBuffer.allocateDirect(PinotByteBuffer.java:36) at com.linkedin.pinot.core.segment.memory.PinotByteBuffer.loadFile(PinotByteBuffer.java:40) at com.linkedin.pinot.core.segment.memory.PinotDataBuffer.loadFile(PinotDataBuffer.java:144) at com.linkedin.pinot.core.segment.store.SingleFileIndexDirectory.mapAndSliceFile(SingleFileIndexDirectory.java:284) at com.linkedin.pinot.core.segment.store.SingleFileIndexDirectory.mapBufferEntries(SingleFileIndexDirectory.java:265) at com.linkedin.pinot.core.segment.store.SingleFileIndexDirectory.load(SingleFileIndexDirectory.java:190) at com.linkedin.pinot.core.segment.store.SingleFileIndexDirectory.(SingleFileIndexDirectory.java:86) at com.linkedin.pinot.core.segment.store.SegmentLocalFSDirectory.loadData(SegmentLocalFSDirectory.java:198) at com.linkedin.pinot.core.segment.store.SegmentLocalFSDirectory.load(SegmentLocalFSDirectory.java:179) at com.linkedin.pinot.core.segment.store.SegmentLocalFSDirectory.(SegmentLocalFSDirectory.java:77) at com.linkedin.pinot.core.segment.store.SegmentDirectory.createFromLocalFS(SegmentDirectory.java:108) at com.linkedin.pinot.core.indexsegment.immutable.ImmutableSegmentLoader.load(ImmutableSegmentLoader.java:108) at com.linkedin.pinot.core.data.manager.realtime.RealtimeTableDataManager.replaceLLSegment(RealtimeTableDataManager.java:291) at com.linkedin.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:279) at com.linkedin.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.downloadSegmentAndReplace(LLRealtimeSegmentDataManager.java:945) at com.linkedin.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.goOnlineFromConsuming(LLRealtimeSegmentDataManager.java:934) at com.linkedin.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromConsuming(SegmentOnlineOfflineStateModelFactory.java:118)

mcvsubbu commented 5 years ago

You are running out of direct memory. You have a few options:

  1. Increase max direct memory for your JVM to suit your needs
  2. Change the load mode to MMAP
  3. Add more servers so that you don't run out of memory