Filtered aggregation fails with IllegalStateException when druid.generic.useDefaultValueForNull=false

[utagaiby]

Indexer fails when using druid.generic.useDefaultValueForNull=false with filtered aggregation

Affected Version

0.18.1

Description

My ingestion spec includes a few filtered aggregations:

{
      "type": "filtered",
      "filter": {
        "type": "and",
        "fields": [{
          "type": "selector",
          "dimension": "dim1",
          "value": "val1"
        },{
          "type": "not",
          "field": {
            "type": "selector",
            "dimension": "dim2",
            "value": "val2"
          }
        }]
      },
      "aggregator": {
        "type": "doubleSum",
        "name": "filtered_aggregation1",
        "fieldName": "Match_Value"
      }
    },
    {
      "type": "filtered",
      "filter": {
        "type": "in",
        "dimension": "dim1",
        "values": ["val3", "val4"]
      },
      "aggregator": {
        "type": "doubleSum",
        "name": "filtered_aggregation2",
        "fieldName": "Match_Value"
      }
    }

The spec works perfectly fine with druid.generic.useDefaultValueForNull=true with filtered aggregations, or with druid.generic.useDefaultValueForNull=false and without filtered aggregations, but when I set druid.generic.useDefaultValueForNull=false and use filtered aggregations in the spec, the ingestion fails with the following error:

org.apache.druid.java.util.common.RE: Error while persisting
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.throwPersistErrorIfExists(AppenderatorImpl.java:228)
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.add(AppenderatorImpl.java:240)
    at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.append(BaseAppenderatorDriver.java:406)
    at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.add(BatchAppenderatorDriver.java:113)
    at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:143)
    at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:945)
    at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:526)
    at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:123)
    at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:421)
    at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:393)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Cannot return double for Null Value
    at org.apache.druid.query.aggregation.NullableNumericAggregator.getDouble(NullableNumericAggregator.java:101)
    at org.apache.druid.query.aggregation.FilteredAggregator.getDouble(FilteredAggregator.java:64)
    at org.apache.druid.segment.incremental.OnheapIncrementalIndex.getMetricDoubleValue(OnheapIncrementalIndex.java:362)
    at org.apache.druid.segment.incremental.IncrementalIndex$DoubleMetricColumnSelector.getDouble(IncrementalIndex.java:1612)
    at org.apache.druid.segment.DoubleColumnSerializerV2.serialize(DoubleColumnSerializerV2.java:120)
    at org.apache.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:498)
    at org.apache.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:192)
    at org.apache.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:920)
    at org.apache.druid.segment.IndexMergerV9.persist(IndexMergerV9.java:785)
    at org.apache.druid.segment.IndexMergerV9.persist(IndexMergerV9.java:754)
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.persistHydrant(AppenderatorImpl.java:1332)
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.access$100(AppenderatorImpl.java:103)
    at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl$3.call(AppenderatorImpl.java:544)
    ... 4 more

[github-actions[bot]]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.