search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.39k stars 3.68k forks source link

"Possible corruption: Memory capacity too small: 47136 < 12845088" exception occur in 0.14.x but not in 0.13.x #8032

Closed quenlang closed 5 years ago

quenlang commented 5 years ago

Hello An exception "Possible corruption: Memory capacity too small: 47136 < 12845088" occurs in the historical log when I performed a query with the Quantiles Sketch in druid 0.14.2 like this

...
    "aggregations": [{
        "type": "quantilesDoublesSketch",
        "name": "resp_time_his",
        "fieldName": "resp_time_his",
        "k": 256
    }],
    "postAggregations": [{
        "type": "quantilesDoublesSketchToQuantiles",
        "name": "respTimeHis",
        "field": {
            "type": "fieldAccess",
            "name": null,
            "fieldName": "resp_time_his"
        },
        "fractions": [0.25, 0.50, 0.75, 0.99]
    }],
...

Then I changed k=256 to k=128, it worked. Also, I downgraded to druid 0.13.0 and set k=256, it worked.

The historical jvm configurations:

-server
-Xms1g
-Xmx3g
-XX:MaxDirectMemorySize=4g
-XX:NewRatio=4
-XX:+UseConcMarkSweepGC
-Duser.timezone=UTC
-Dfile.encoding=UTF-8
-Djava.io.tmpdir=var/tmp
-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager

The historical runtime configurations:

druid.service=druid/historical
druid.port=8083
druid.server.http.numThreads=25
druid.processing.buffer.sizeBytes=200000000
druid.processing.numThreads=4
druid.processing.numMergeBuffers=1
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":130000000000}]
druid.server.maxSize=130000000000

And the dataset which I queryed is very small, I have no idea of this exception. @himanshug Can you give me some advices? Thank you ! Have a nice day!

The detail error messag:

2019-07-05T05:58:57,982 ERROR [qtp700171153-70[groupBy_[SVR_ACTION_DATA_MIN]_d149d0bf-6bb7-4569-bbbc-43ebb5567022]] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class com.yahoo.sketches.SketchesArgumentException, exceptionMessage=Possible corruption: Memory capacity too small: 47136 < 12845088, exception=com.yahoo.sketches.SketchesArgumentException: Possible corruption: Memory capacity too small: 47136 < 12845088, query=GroupByQuery{dataSource='SVR_ACTION_DATA_MIN', querySegmentSpec=MultipleSpecificSegmentSpec{descriptors=[SegmentDescriptor{interval=2019-07-04T08:05:00.000Z/2019-07-04T08:35:00.000Z, version='2019-07-04T08:01:31.683Z', partitionNumber=0}, SegmentDescriptor{interval=2019-07-04T08:05:00.000Z/2019-07-04T08:35:00.000Z, version='2019-07-04T08:01:31.683Z', partitionNumber=1}]}, virtualColumns=[], limitSpec=DefaultLimitSpec{columns='[]', limit=2147483647}, dimFilter=(biz_system_id = 1150 && data_type = APP), granularity=AllGranularity, dimensions=[DefaultDimensionSpec{dimension='biz_system_id', outputName='bizSystemId', outputType='STRING'}], aggregatorSpecs=[DoublesSketchAggregatorFactory{name=resp_time_his, fieldName=resp_time_his, k=256}], postAggregatorSpecs=[DoublesSketchToQuantilesPostAggregator{name='respTimeHis', field=FieldAccessPostAggregator{name='null', fieldName='resp_time_his'}, fractions=[0.2199999988079071, 0.550000011920929, 0.9800000190734863, 0.9900000095367432]}], havingSpec=null}, peer=10.128.1.108}
com.yahoo.sketches.SketchesArgumentException: Possible corruption: Memory capacity too small: 47136 < 12845088
    at com.yahoo.sketches.quantiles.DirectUpdateDoublesSketchR.checkDirectMemCapacity(DirectUpdateDoublesSketchR.java:197) ~[?:?]
    at com.yahoo.sketches.quantiles.DirectUpdateDoublesSketch.wrapInstance(DirectUpdateDoublesSketch.java:108) ~[?:?]
    at com.yahoo.sketches.quantiles.DoublesUnionImpl.updateLogic(DoublesUnionImpl.java:194) ~[?:?]
    at com.yahoo.sketches.quantiles.DoublesUnionImpl.update(DoublesUnionImpl.java:104) ~[?:?]
    at org.apache.druid.query.aggregation.datasketches.quantiles.DoublesSketchMergeBufferAggregator.aggregate(DoublesSketchMergeBufferAggregator.java:70) ~[?:?]
    at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.aggregate(AbstractBufferHashGrouper.java:165) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.SpillingGrouper.aggregate(SpillingGrouper.java:167) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.ConcurrentGrouper.aggregate(ConcurrentGrouper.java:267) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.Grouper.aggregate(Grouper.java:82) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$1.accumulate(RowBasedGrouperHelper.java:270) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$1.accumulate(RowBasedGrouperHelper.java:247) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.ConcatSequence.lambda$accumulate$0(ConcatSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.FilteringAccumulator.accumulate(FilteringAccumulator.java:41) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.ConcatSequence.accumulate(ConcatSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:82) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:168) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:45) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:148) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-core-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:244) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:232) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_60]
    at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247) ~[druid-processing-0.14.2-incubating.jar:0.14.2-incubating]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_60]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
gianm commented 5 years ago

@AlexanderSaydakov / @leerho does this look like a potential issue to you? druid-0.14.2 is using sketches-core 0.13.3, and druid-0.13.0 is using sketches-core 0.10.3.

AlexanderSaydakov commented 5 years ago

What is the type of the input field "fieldName": "resp_time_his"?

quenlang commented 5 years ago

@gianm @AlexanderSaydakov Thanks for quick reply. The type of resp_time_his is sketche object. We build outside of druid with sketchs-core 0.10.3.

AlexanderSaydakov commented 5 years ago

What k do the input sketches have? I could not reproduce this so far. I would appreciate some simple reproducible scenario.

quenlang commented 5 years ago

k=256.