Cant using Date Histogram aggregation Average Bucket with nested field

[yechanpark]

Plugin version: 6.4.1-1.0.0 Kibana version: 6.4.1

hi. i want to aggregate nested fields day by day using Average Bucket. it causing error.

following is mapping

{
  "mapping": {
    "fluentd": {
      "properties": {
        "PERF": {
          "type": "long"
        },
        "REQ_PATH": {
          "type": "keyword"
        },
        "REQ_TIME": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss"
        },
        "profiles": {
          "type": "nested",
          "properties": {
            "PERF": {
              "type": "integer"
            },
            "REQ_TIME": {
              "type": "date",
              "format": "yyyy-MM-dd HH:mm:ss"
            },
            "caller": {
              "type": "keyword"
            }
          }
        }
      }
    }
  }
}

1. Cant aggregate nested fields by Date Histogram with nested date field using Average Bucket following is my visualize setting. visualize type is 'horizontal bar'.

i want to aggregate 'profiles.PERF' average day by day. so i made Average Bucket in Metrics's Y-Axis, and using Date Histogram and Interval is 'Daily'. Date Histogram Field is 'profiles.REQ_TIME'. i can using 'doc.REQ_TIME' too. it doesn't matter.

i want to aggregate by each 'profiles.caller' term . so, i made Buckets's X-Axis, and using 'profiles.caller' Terms. and i want to ordering each caller by Alphabetical ASC.

when i clicked 'Apply Changes' button, error had occured. following is error message in kibana.

Request to Elasticsearch failed: 
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "buckets_path aggregation does not exist for aggregation [1]: 1-bucket>1-metric"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "myindex",
        "node": "mynode",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "buckets_path aggregation does not exist for aggregation [1]: 1-bucket>1-metric"
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "buckets_path aggregation does not exist for aggregation [1]: 1-bucket>1-metric",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "buckets_path aggregation does not exist for aggregation [1]: 1-bucket>1-metric"
      }
    }
  },
  "status": 400
}

i want to show my information graph like this

2. alphabetical ordering bug when using split-series instead using Average Bucket) i found the a contingency plan (using Y-Axis's split-series instead X-Axis's Average Bucket), but there is a alphabetical ordering bug. (i don't know exactly that's a bug. i guess it can be kibana's official functionality about 'split-series')

following is my visualize setting. visualize type is 'horizontal bar'.

i want to aggregate 'profiles.PERF' average day by day. but i cant using Average Bucket now. so, i made Metrics's Y-Axis and set profiles.PERF's average only.

i want to aggregate by each 'profiles.caller' term . so, i made Buckets's X-Axis, and using 'profiles.caller' Terms. and i want to ordering each caller by Alphabetical ASC.

plus, i made Buckets's Split Series and set Sub Aggregation is Date Histogram for aggregate day by day. and set profles.REQ_TIME field.

and i clicked 'Apply Changes' and graph has shown like this

but, when i selected particular date, it is working properly like this.

2 image_models are exist in 2018-11-30 docs only. not exist in 2018-11-29 docs.

3. third try actually, im using upper case graph. following is my setting

and there are 4 Y-Axis (today, yesterday, 2 and 3 days ago) like this

and following is buckets

and following is actually i wanted graph. graph is perfectly working as i intended.

differences are 1,2 cases are using nested field (all of fields are nested include date) and using date histogram for aggregate day by day 3 case is using non-nested fields (all of fields are non-nested include date) and using date range for aggregate day by day

can i drawing graph as i intended in nested world?

i'm waiting response.

---

i guess here is a problem.

when Metric's Aggregation field setted as 'profiles.PERF', error has occured. but Metric's Aggragation field setted as 'doc.PERF', there isn't error.

all of mixed cases are do not working as i intended. (both 'Date Histogram' and 'Date Range') doc.REQ_TIME + profiles.perf = error occured profiles.REQ_TIME + profiles.perf = error occured doc.REQ_TIME + doc.PERF = show line, but i don't want this value profiles.REQ_TIME + doc.PERF = no error, but doesn't show line

and following is elasticsearch's java exception message. [2018-12-03T11:48:34,762][DEBUG][o.e.a.s.TransportSearchAction] [host] All shards failed for phase: [query] org.elasticsearch.ElasticsearchException$1: No aggregation [7-metric] found for path [7-bucket>7-metric] at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:657) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:131) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:48) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.InitialSearchPhase$2.lambda$onFailure$1(InitialSearchPhase.java:222) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.InitialSearchPhase.maybeFork(InitialSearchPhase.java:176) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.InitialSearchPhase.access$000(InitialSearchPhase.java:48) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.InitialSearchPhase$2.onFailure(InitialSearchPhase.java:222) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:526) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1068) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1165) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1149) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:66) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.action.search.SearchTransportService$6$1.onFailure(SearchTransportService.java:384) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService$2.onFailure(SearchService.java:341) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.1.jar:6.4.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152] Caused by: java.lang.IllegalArgumentException: No aggregation [7-metric] found for path [7-bucket>7-metric] at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.resolvePipelineAggregatorOrder(AggregatorFactories.java:401) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.resolvePipelineAggregatorOrder(AggregatorFactories.java:351) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:324) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.AggregatorFactory.<init>(AggregatorFactory.java:187) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.<init>(ValuesSourceAggregatorFactory.java:40) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregatorFactory.<init>(TermsAggregatorFactory.java:73) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.bucket.terms.TermsAggregationBuilder.innerBuild(TermsAggregationBuilder.java:338) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:311) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:329) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService.parseSource(SearchService.java:766) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService.createContext(SearchService.java:575) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:551) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:347) ~[elasticsearch-6.4.1.jar:6.4.1] at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.4.1.jar:6.4.1] ... 9 more

[ppadovani]

I believe this is a duplicate of #90 . The way Kibana handles index patterns makes supporting nested time fields extremely difficult. Additionally, if you have a nested object array with multiple objects each with a different date, how is the overall document ordered with respect to other matched documents?

For example: Doc A: time: 1/1/2000 nested: time 2: 1/4/2000 time 3: 1/6/2000

Doc B: time: 1/2/2000 nested: time 2: 1/2/2000 time 3: 1/7/2000

What order should the documents be returned? By the parent time? The underlying document format and index structure of Elasticsearch make ordering the documents in a logical way difficult, and likely would require post-processing to accomplish. In a regular RDBMS, this would be a relatively trivial problem to solve.

[yechanpark]

@ppadovani thx for your response. I understood about difficult to implementation in kibana.

i thought 'elasticsearch can handle nested aggregation, so kibana maybe can handle this issue'

actually, i handled this issue via built-in Vega in Kibana 6.4. (not kibana's traditional visualization) but there are some critical issues like performence (can not load data dinamically).

so, I gave up using kibana.. 😢

but still your plugin can handle many graphs ! very useful 😄

thanks!