search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.49k stars 24.88k forks source link

Add support for ENRICH in ESQL CsvTests #101424

Open luigidellaquila opened 1 year ago

luigidellaquila commented 1 year ago

CsvTests can now support the planning of ENRICH command, but the execution is still missing. Since CsvTests do not use ES indices, we will have to mock the policy execution.

The problem: CsvTests don't use ES storage, they just load data from CSV and mock the data load phase. ENRICH is all based on an enrich index, so we need an actual ES instance running to properly test it, or a way to mock both the loading of the data AND the enrich processor, to pick data from CSV (but does it make sense at all? Is it worth having tests on a completely mocked feature?)

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-ql (Team:QL)

elasticsearchmachine commented 1 year ago

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

getkub commented 1 year ago

A great idea. In large organisation we need to do enrichment from CSV directly. Don't want to complicate using 'enrich policy' etc. As it in Splunk, the on-the-fly enrich using lookup/csv file is greatly appreciated

elasticsearchmachine commented 11 months ago

Pinging @elastic/es-analytics-geo (Team:Analytics)

alex-spies commented 9 months ago

Since @nik9000 improved the query execution to not need a search service, but a mere SearchExecutionContext, we could have a better chance of running the Enrich tests without whole integration tests. Not sure if that's enough, but maybe basing our tests on MapperServiceTestCase and having more "proper" indices would help.

I made a (quick and dirty) prototype a while ago that does that, though it doesn't do any enriching.