search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.62k stars 24.64k forks source link

ES|QL: Allow operations on non-existing fields #112912

Open dgieselaar opened 1 week ago

dgieselaar commented 1 week ago

Description

As an engineer, I sometimes have to query a set of indices, where I'm not sure if a field exists in those indices, but I also don't care. E.g., I might want to group by service.name and service.environment in logs-*, but the user might only have service.name. I want to signal to the _query endpoint that it's OK if service.environment does not exist. Currently this fails:

FROM logs-* | STATS BY service.name, service.environment

per @not-napoleon's suggestion, we could do something like this:

FROM logs-* | STATS BY service.name, IF_EXISTS(service.environment)

I would like it to work for STATS but also things like WHERE, EVAL, etc.

elasticsearchmachine commented 1 week ago

Pinging @elastic/es-analytical-engine (Team:Analytics)

nik9000 commented 1 week ago

I wonder if IF_EXISTS could be a function that's a noop if the field exists and null if it doesn't.

astefan commented 1 week ago

EQL has "optional fields": https://www.elastic.co/guide/en/elasticsearch/reference/current/eql-syntax.html#eql-syntax-optional-fields

dgieselaar commented 1 week ago

@astefan IMHO a syntax like that (I assume this conflicts with params) would be ideal (over a function)