ESQL: If query lists many many many many fields then the internal field caps action will fail

[nik9000]

Description

This is what the error message looks like:

Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: too_complex_to_determinize_exception: Determinizing automaton with 2254 states and 2347 transitions would require more than 10000 effort.
    at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:735)
    at org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:72)
    at org.apache.lucene.util.automaton.CharacterRunAutomaton.<init>(CharacterRunAutomaton.java:36)
    at org.apache.lucene.util.automaton.CharacterRunAutomaton.<init>(CharacterRunAutomaton.java:23)
    at org.elasticsearch.common.regex.Regex.simpleMatcher(Regex.java:128)
    at org.elasticsearch.action.fieldcaps.TransportFieldCapabilitiesAction$NodeTransportHandler.lambda$messageReceived$0(TransportFieldCapabilitiesAction.java:541)
    at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:348)

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[nik9000]

I need to confirm this. I'm getting the error message from a log but it really look like someone did something like FROM * | KEEP a, b, c, d, ....., a1, b2,.....d1324.

[nik9000]

This reproduces it:

curl -uelastic:password -XDELETE localhost:9200/test
curl -uelastic:password -HContent-Type:application/json -XPOST localhost:9200/test/_doc -d'{"foo": "bar"}'

echo '{"query": "FROM test' > /tmp/query
for i in $(seq 1 300); do
    echo -n '| KEEP f0'$i >> /tmp/query
    for j in $(seq 1 300); do
        echo -n ', f'$j$i >> /tmp/query
    done
    echo >> /tmp/query 
done
echo '"}' >> /tmp/query
curl -uelastic:password -HContent-Type:application/json -XPOST localhost:9200/_query -d@/tmp/query

Determinizing automaton with 691104 states and 773832 transitions would require more than 10000 effort.

[nik9000]

The numbers I used above are quite high to reproduce. This'll do it with much fewer fields:

curl -uelastic:password -XDELETE localhost:9200/test
curl -uelastic:password -HContent-Type:application/json -XPOST localhost:9200/test/_doc -d'{"foo": "bar"}'

echo '{"query": "FROM test' > /tmp/query
for i in $(seq 1 42); do
    echo -n '| KEEP f0'$i >> /tmp/query
    for j in $(seq 1 300); do
        echo -n ', f'$j$i >> /tmp/query
    done
    echo >> /tmp/query 
done
echo '"}' >> /tmp/query
curl -uelastic:password -HContent-Type:application/json -XPOST localhost:9200/_query -d@/tmp/query

[nik9000]

We can make those examples work.

I think it's possible that there are other ways to cause this, so there's more to push here. I think.

[nik9000]

I think we should do a few things:

• [ ] Make sure field caps itself sends back a 400 error when it can't determinize the pattern. Something saying "too complex" or something. This has never worked, but these days it's good to make sure the error has a 400 code. I mean, it was always good, but it matters more to us now.
• [ ] Make sure ESQL returns a similar 400 error when field caps can't determinize the pattern. This should probably be similar to the "your ESQL is too deep" error message, though we should make it clear it's because the fields were too complex.
• [ ] Make sure that any individual field pattern that super complex will also make a 400 instead of a 500.
• [ ] We should figure out why field caps needs to send every field twice - one like field and once like field*. I know there's a reason, but it's expensive.
• [ ] Presuming we still want to do that, could try and make field caps better able to handle that. I think it'd be pretty easy to detect the foo* case and group those into their own list and build them with makeStringUnion and append a single * pattern after those. I don't know what that'd do for the determinization process, but it sure feels like it'd be a lot kinder.