"Invalid aggregator order path [1]. Unknown aggregation [1]" on reverse nested aggregation

[dtheodor]

I'm trying to create some graphs with terms aggregations where the terms belong to nested documents and aggregated metrics go back to the original document (by clicking "Count from parent document" in the UI). Kibana is generating invalid queries that result in the following failure:

"reason": {
  "type": "aggregation_execution_exception",
  "reason": "Invalid aggregator order path [1]. Unknown aggregation [1]"  
}

My data model has the following structure

{
  "file": {
    // not nested
    "size": 52
  },
  "access": [{
    // this is the nested document
    "type": "a type"
  }]
}

and I'm trying to graph the sum(size) per access type (SELECT sum(file.size), access.type GROUP BY access.type if you want)

This is the invalid query generated that results in the error above

{
    "aggs": {
        "nested_2": {
            "nested": {
                "path": "access"
            },
            "aggs": {
                "2": {
                    "terms": {
                        "field": "access.type",
                        "size": 5,
                        "order": {
                            "1": "desc"
                        }
                    },
                    "aggs": {
                        "nested_1": {
                            "reverse_nested": {},
                            "aggs": {
                                "1": {
                                    "sum": {
                                        "field": "file.size"
                                    }
                                }
                            }
                        },
                        "count_2": {
                            "reverse_nested": {}
                        }
                    }
                }
            }
        }
    },
}

The following is the fixed query that should have been generated. The only difference is in the "order" parameter specification, changed from "order": {"1": "desc"} to "order": {"nested_1>1": "desc"}

{
    "aggs": {
        "nested_2": {
            "nested": {
                "path": "access"
            },
            "aggs": {
                "2": {
                    "terms": {
                        "field": "access.type",
                        "size": 5,
                        "order": {
                            "nested_1>1": "desc"
                        }
                    },
                    "aggs": {
                        "nested_1": {
                            "reverse_nested": {},
                            "aggs": {
                                "1": {
                                    "sum": {
                                        "field": "file.size"
                                    }
                                }
                            }
                        },
                        "count_2": {
                            "reverse_nested": {}
                        }
                    }
                }
            }
        }
    },
}

[ppadovani]

Can you please indicate the Kibana version? This affects which branch I fix first. Until I get word back I'm going to assume 6.2.2.

Thanks!

Pierre

[dtheodor]

Yes, this is the 6.2.2 version

[ppadovani]

I believe this is now fixed in the 6.2.2.-1.01 release that I just pushed out.

Please verify, and update this issue.

Please do not close this issue if verified, as I will close this once I have back ported the fix to all versions containing the issue.

[dtheodor]

Yes, this query runs successfully with the 6.2.2.-1.01 release.

I have discovered another related issue, not sure if it was introduced with this change or already existed.

I'm running the following terms aggregation on a nested object:

SELECT count(distinct access.member.email),
access.member.domain
GROUP BY access.member.domain

The data model is

{
  "access": [{
    // this is a nested document
    "members:" [{
      // also a nested document (double nesting)
      "domain": "...",
      "email": "..."
  }]
}

So the query is touching only the "access.member" nested document, both the term and the metric are on the same nested document.

I'm getting an error from the following bogus query:

{
    "reason": {
        "type": "aggregation_execution_exception",
        "reason": "Invalid aggregator order path [nested_1>1]. Unknown aggregation [nested_1]"
    }
}

{
    "aggs": {
        "nested_2": {
            "nested": {
                "path": "access.members"
            },
            "aggs": {
                "2": {
                    "terms": {
                        "field": "access.members.domain",
                        "size": 5,
                        "order": {
                            "nested_1>1": "desc"
                        }
                    },
                    "aggs": {
                        "1": {
                            "cardinality": {
                                "field": "access.members.email"
                            }
                        }
                    }
                }
            }
        }
    },
}

There's no nested_1, the order in this case should have just been "order": {"1": "desc"}

[dtheodor]

I just tried this 2nd query with the previous release (6.2.2.-1.0.0) , and the the correct query is generated. So this is a new issue introduced in this latest release.

The correct generated query in 6.2.2.-1.0.0:

{
    "aggs": {
        "nested_2": {
            "nested": {
                "path": "access.members"
            },
            "aggs": {
                "2": {
                    "terms": {
                        "field": "access.members.domain",
                        "size": 5,
                        "order": {
                            "1": "desc"
                        }
                    },
                    "aggs": {
                        "1": {
                            "cardinality": {
                                "field": "access.members.email"
                            }
                        }
                    }
                }
            }
        }
    }
}

[ppadovani]

Yep, looks like the fix needed is going to be somewhat more involved than I had initially anticipated. This may take a few more days to get working correctly.

[ppadovani]

Ok... I've got another build for you to try. I've built this, but it isn't an official release even though there is a version bump.

https://github.com/ppadovani/KibanaNestedSupportPlugin/releases/download/6.2.2-1.0.1/nested-fields-support-6.2.2-1.0.2.zip

If you report this as fixing the issue, then I'll cut this, and publish. If this doesn't fix the issue, then I'm going to modify my test data generation scripts to add structure and data that better aligns with the issue you are seeing then work through a solution.

[dtheodor]

Looks like the latest release has fixed the queries I'm trying, haven't been able to run into any issues yet

[ppadovani]

I'll hold off cutting this, and work on #55 as part of the same release.

Thanks for the bug reports!