ESQL: Safeguards against huge transport requests - Githubissues

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine

https://www.elastic.co/products/elasticsearch

Other

1.17k stars 24.84k forks source link

ESQL: Safeguards against huge transport requests #112873

Open alex-spies opened 2 months ago

alex-spies commented 2 months ago

Executing ESQL queries normally requires that the coordinator node and data nodes communicate: the coordinator sends logical plans to the data nodes, the data nodes send pages with results back to the coordinator.

In both directions, the transport message size seems to be unbounded, and there also seems to be no circuit breaker; we've seen cases where particularly large logical plans caused gigabytes of data to be in buffered in the NettyAllocator.

While some issues were addressed in https://github.com/elastic/elasticsearch/pull/112008, https://github.com/elastic/elasticsearch/pull/111447 and https://github.com/elastic/elasticsearch/pull/111973, we should find other situations where this can happen, test it and fix it if needed. I.e.

Try to provoke huge LogicalPlans in other ways and test this.
Try to provoke huge pages to be sent from data nodes, e.g. super many columns, or with individual values that are huge etc.

This is similar to our HeapAttack tests, but distributed.

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-analytical-engine (Team:Analytics)