Script compilation limiting should be user actionable

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine

https://www.elastic.co/products/elasticsearch

Other

1.23k stars 24.85k forks source link

Script compilation limiting should be user actionable #55684

Open stu-elastic opened 4 years ago

stu-elastic commented 4 years ago

Script compilation limiting frequently occurs on ingest pipelines and can occur during shard migrations, as well as other automatic actions.

We want scripts to be properly parameterized to avoid unnecessary compilations, but users are default hitting limits with properly parameterized scripts.

How can we protect ES without blocking users from using scripts?

cc: @HonzaKral

elasticmachine commented 4 years ago

Pinging @elastic/es-core-infra (:Core/Infra/Scripting)

stu-elastic commented 4 years ago

From discussion in slack:

I'd honestly go about it the other way - just flag those script uses that are problematic and safe to enforce:
* triggered by user action (so that error can be surfaced)
* not using stored scripts (including creation of stored scripts)
* exclude "debugging" APIs (watcher/watch/_execute , _ingest/pipeline/_simulate , _scripts/painless/_execute, ...)
Only inline scripts in _search come to mind rn that fit all the categories

stu-elastic commented 3 years ago

1) Better error message for thrashing the cache 2) Internal systems that store scripts which have their own cache should not be rate-limited. TODO: check watcher, this is the case with ingest.

Maybe exempt services that do their caching from global cache.