elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.43k stars 24.57k forks source link

ESQL: union types not treating widened data types correctly #111277

Open alex-spies opened 1 month ago

alex-spies commented 1 month ago

Union types allow to use differently typed fields from different indices if they have the same name, e.g. client_ip with type ip in index1 and client_ip with type keyword in index2.

However, this seems broken for data types that esql widens, e.g. a field mapped short which esql internally represents using integer.

Reproducer:

curl -u elastic:password -H "Content-Type: application/json" "127.0.0.1:9200/testidx" -XPUT -d '{"mappings": {"properties": {"a": {"type":"short"}}}}' 

curl -u elastic:password -H "Content-Type: application/json" "127.0.0.1:9200/testidx2" -XPUT -d '{"mappings": {"properties": {"a": {"type":"integer"}}}}'

curl -u elastic-admin:elastic-password -H "Content-Type: application/json" "127.0.0.1:9200/testidx/_doc" -d '{"a": 1}'

curl -u elastic-admin:elastic-password -H "Content-Type: application/json" "127.0.0.1:9200/testidx2/_doc" -d '{"a": 1}' 

FROM testidx*

FROM testidx* | EVAL a = a::integer

The second query should work, but it returns an error complaining about a being ambiguous due to being mapped to different types.

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-analytical-engine (Team:Analytics)

alex-spies commented 1 month ago

https://github.com/elastic/elasticsearch/pull/111253 seems to solve this problem because it makes it so that we never interact with the non-widened data types; e.g. if a field has type short we'd always treat it as integer and the reproducer from above works just fine.

alex-spies commented 2 weeks ago

Another special case we bumped into: TO_STRING(field) where field is float in one index but double in another, c.f. https://github.com/elastic/elasticsearch/issues/111910