Closed quenlang closed 5 years ago
I found a lot of Allocating new littleEndByteBuf
operations in the log.
Finally, allocated 333023 littleEndByteBuf and the usage of direct memory arrived 18GB, and OS killed this task.
The ingested data was very small, just 2GB in six hours, and all the dimensions have very low cardinality.
Why the task need so many direct memory?
@himanshug
Can you give me some advices? Thank you so much!
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,020]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,021]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,022]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,023]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,024]
2019-06-14T06:24:34,445 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,025]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,026]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,027]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,028]
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,029]
Unfortunately, It is very hard to offer concrete advice on these things, I can give some general points for you to further investigate.
OS killed this task
I am assuming that means linux OOM killer killed the process. If yes, then it probably can't be the memory used to load the segments in page cache as that doesn't count towards anonymous memory usage. So, that means the process really allocated loads of direct memory using those buffers you mentioned.
There could be a genuine memory leak there. You can identify that by doing a manual GC (e.g. using "jcmd
Check whether segment handoffs are working fine, you can add "pushTimeout" setting from above if need be.
you might be hitting issue described in https://github.com/apache/incubator-druid/pull/6699 , if you can build druid with that patch applied then try that.
That said, please attache the task log of the process that ends up dead due to too much memory usage. I will take a quick look to see if there is something obvious that shows up.
HTH.
@himanshug Thank you so much for the quick replay!
I had performed a manual GC with using jcmd GC.run
, but did not free up the direct memory.
The task log of the process that ends up dead due to too much memory usage as below.
Thanks !
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6865}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6866}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6867}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6868}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6869}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6870}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6871}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6872}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6873}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6874}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6875}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6876}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6877}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6878}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6879}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6880}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6881}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6882}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6883}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6884}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6885}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6886}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6887}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6888}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6889}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6890}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6891}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6892}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6893}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6894}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6895}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6896}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6897}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6898}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6899}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6900}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6901}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6902}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6903}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6904}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6905}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6906}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6907}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6908}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6909}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6910}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6911}]
2019-06-14T06:24:10,907 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6912}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6913}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6914}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6915}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6916}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6917}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6918}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6919}]
2019-06-14T06:24:10,908 INFO [appenderator_merge_0] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Adding hydrant[FireHydrant{, queryable=ZJK_TEST_2019-06-13T06:00:00.000Z_2019-06-13T12:00:00.000Z_2019-06-14T06:17:24.368Z, count=6920}]
2019-06-14T06:24:11,090 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Using SegmentWriteOutMediumFactory[TmpFileSegmentWriteOutMediumFactory]
2019-06-14T06:24:11,091 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Completed version.bin in 0 millis.
2019-06-14T06:24:11,091 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Completed factory.json in 0 millis
2019-06-14T06:24:11,415 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[20]
2019-06-14T06:24:11,416 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[12]
2019-06-14T06:24:11,419 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[4]
2019-06-14T06:24:11,419 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[4]
2019-06-14T06:24:11,419 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[4]
2019-06-14T06:24:13,376 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[168]
2019-06-14T06:24:13,376 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[168]
2019-06-14T06:24:13,376 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMerger - Allocating dictionary merging direct buffer with size[64]
****************** many many many Allocating dictionary merging direct buffer with size [*] *******************
2019-06-14T06:24:13,987 INFO [appenderator_merge_0] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[uri] conversions with cardinality[1,084] in 722 millis.
2019-06-14T06:24:13,988 INFO [appenderator_merge_0] org.apache.druid.segment.IndexMergerV9 - Completed dim conversions in 2,596 millis.
2019-06-14T06:24:20,682 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[53]
2019-06-14T06:24:20,682 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[54]
2019-06-14T06:24:20,683 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[55]
2019-06-14T06:24:20,683 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[56]
2019-06-14T06:24:20,683 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[57]
2019-06-14T06:24:20,683 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[58]
2019-06-14T06:24:20,683 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[59]
2019-06-14T06:24:20,683 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[60]
****************** many many many Allocating new littleEndByteBuf [*] until 333,029 *******************
2019-06-14T06:24:34,446 INFO [appenderator_merge_0] org.apache.druid.segment.CompressedPools - Allocating new littleEndByteBuf[333,029]
so there were 6920 segments to be merged together with 80 columns each, merging that does need alot of those buffers, about 6920*80 . I think there are too many intermediate persists because of maxBytesInMemory
which in your case , by default, is 2g/6 = 341M
. OnHeapIncrementalIndex does overestimate the size due to sketches in there and might be persisting segments too aggressively.
Here is what I would try out...
set maxBytesInMemory = Integer.MAX_VALUE (to effectively disable it)
adjust maxRowsInMemory
(default 1000000) to a lower value or increase Xmx on task process so that it doesn't get into too much of GC . You might have to adjust druid.worker.capacity=10
to a lower value if Xmx was increased. Now find the right balance of maxRowsInMemory
and Xmx
on task process so that you don't end up with thousands of segments to merge in the end but much lower number. That will drive down the off-heap buffer allocation .
(Higher maxRowsInMemory
means lower the number of segments to merge in the end but higher jvm heap utilization )
@himanshug Thank you so much! I had set maxBytesInMemory = Integer.MAX_VALUE in my task and the number of segments reduce to 1500. Finally, the task process works very well. direct memory used 4GB(originally used almost 20GB) when the process handoff segments. Thank you! Have a good day!
likely related #6743
https://github.com/apache/incubator-druid/pull/7919 should further reduce the memory utilization . however that PR is independent and I am closing this bug.
Hi, all I found a problem when i use kafka index service. The task process uses a lot of direct memory when publishing segments, almost 6x than ingesting data stage. I'm using druid-0.13.0.
At ingesting time,
RES
of the task is 1.647g.And the
heap
、non-heap
、direct
are 221M、120M and 864MBut at publishing time,
RES
of the task is 6.046gAnd the
heap
、non-heap
、direct
are 266M、124M and 5204MThe direct memory increase 864M to 5204M when publishing begins. How can i control the usage of direct memory?
Any suggestion? Thanks!
middleManager runtime.properties
Datasource has 35 dimensions and 45 metrics. The spec file