Composite terms aggregation fails on aliased indexes with dynamic templates

[c-dante]

Using docker image: docker.elastic.co/elasticsearch/elasticsearch:7.1.1

Elasticsearch version (bin/elasticsearch --version): 7.1.1

Plugins installed: []

JVM version (java -version):

OS version (uname -a if on a Unix-like system): Linux amp 5.0.13-arch1-1-ARCH #1 SMP PREEMPT Sun May 5 18:05:41 UTC 2019 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior: Expected: Running a composite aggregate with after on dynamic, alias'd indexes don't fail on undefined fields.

Actual: If an aliased index is missing a field, it crashes with number format excetion

Steps to reproduce: Here's a gist with the set up + logs https://gist.github.com/c-dante/9c973425f0dfd6ed3b22ec0c538e03e8

Spark notes of the process:

1. Make an index template and alias with dynamic field templates: (_templates/my-idx)
2. Have 1 index in the alias be empty (never sees the dynamic fields): (/my-idx-default)
3. Have another index have fields and documents: (/my-idx-content)
4. A composite aggregate on the outer index: (post to /my-idx)
5. Run the same aggregate with after, it crashes with number format.

This looks to be a further case of https://discuss.elastic.co/t/rollup-job-numberformatexception/178195

Provide logs (if relevant): Here's a shorter snippet of the logs, gist has full from request

{"type": "server", "timestamp": "2019-06-20T21:41:31,046+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "docker-cluster", "node.name": "a510efed5d96", "cluster.uuid": "YkYTpw71QJyC6JxUx5rhwA", "node.id": "LMPLncMjR9uEuQWxX9h9iQ",  "message": "[my-idx-default][4], node[LMPLncMjR9uEuQWxX9h9iQ], [P], s[STARTED], a[id=eT9ydoflR-KYQnQz6dRQ3Q]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[my-idx], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={\"size\":0,\"aggregations\":{\"test\":{\"composite\":{\"size\":1,\"sources\":[{\"k_0\":{\"terms\":{\"field\":\"k_0\",\"missing_bucket\":false,\"order\":\"asc\"}}}],\"after\":{\"k_0\":\"apple\"}}}}}}] lastShard [true]" , 
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [a510efed5d96][172.17.0.2:9300][indices:data/read/search[phase/query]]",
"Caused by: java.lang.NumberFormatException: For input string: \"apple\"",
"at jdk.internal.math.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2054) ~[?:?]",
"at jdk.internal.math.FloatingDecimal.parseDouble(FloatingDecimal.java:110) ~[?:?]",
"at java.lang.Double.parseDouble(Double.java:549) ~[?:?]",
"at org.elasticsearch.search.DocValueFormat$1.parseLong(DocValueFormat.java:119) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at org.elasticsearch.search.aggregations.bucket.composite.LongValuesSource.setAfter(LongValuesSource.java:154) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesCollectorQueue.<init>(CompositeValuesCollectorQueue.java:86) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator.<init>(CompositeAggregator.java:94) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationFactory.createInternal(CompositeAggregationFactory.java:49) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:217) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:214) ~[elasticsearch-7.1.1.jar:7.1.1]",
"at ...

And the response:

{
    "took": 13,
    "timed_out": false,
    "_shards": {
        "total": 10,
        "successful": 5,
        "skipped": 0,
        "failed": 5,
        "failures": [
            {
                "shard": 0,
                "index": "my-idx-default",
                "node": "LMPLncMjR9uEuQWxX9h9iQ",
                "reason": {
                    "type": "number_format_exception",
                    "reason": "For input string: \"apple\""
                }
            }
        ]
    },
    "hits": {
        "total": {
            "value": 4,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    },
    "aggregations": {
        "test": {
            "after_key": {
                "k_0": "banana"
            },
            "buckets": [
                {
                    "key": {
                        "k_0": "banana"
                    },
                    "doc_count": 1
                }
            ]
        }
    }
}

[jimczi]

Thanks for reporting @c-dante , this issue is already fixed by https://github.com/elastic/elasticsearch/pull/41280 that assumes that any unmapped field is of type keyword (and not numeric as it is today).

[c-dante]

Cool -- saw in that thread that it should have made it into 7.1.x and the timing lined up commit wise, but I see the 7.2.0 tag now.