search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.49k stars 24.88k forks source link

ESQL: Copy filters through DISSECT and GROK #113115

Open nik9000 opened 2 months ago

nik9000 commented 2 months ago

Description

It should be possible to push a filter "through" DISSECT or GROK. Like so:

FROM foo
| DISSECT message "login failed %{user} at %{ip}"
| WHERE user == "nik9000"

Right now we have to scan everything but we know that this can only match if message starts with login failed nik9000 at. That requires a fair bit of brains about how DISSECT works. At worst, we know that message contains nik9000.

There are other cases like:

FROM foo
| DISSECT message "%{ip} - login failed %{other1} %{user} %{other2}"
| WHERE user == "nik9000"

where we really can only be sure that the message contained nik9000. It'd be lovely if we could extract a query from this. It wouldn't be sure that the document matches, but we still have the rest of ESQL - in this case instead of "pushing" the filter down we could simply copy it as a first pass and let ESQL's mechanisms double check.

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-analytical-engine (Team:Analytics)