[Transform][Search] Meta Issue: improve Transform robustness in case of search errors

[hendrikmuhs]

Based on recent feedback we discovered some shortcomings in transforms when search experiences a problem. This meta issue lists improvements.

This is not a transform issue, but covers search, which builds the foundation for transform.

/CC @dnhatn, @javanna, @przemekwitek, @wchaparro, @droberts195, @sophiec20, @dimkots

My suggestions, feel free to add yours. I suggest to spawn new issues for every task and discuss the details there, to keep this meta issues clean.

• [ ] point in time breaks on shard relocation, @dnhatn is already looking into this
• [ ] search fails with a QueryShardException, this exception type classifies itself as BAD_REQUEST, but not every QueryShardException is a BAD_REQUEST. A bad request is considered an irrecoverable error, that's why transform decides not to retry(except it is configured as unattended)
• [ ] SearchExecutionContext treats a generic Exception as QueryShardException
  • [ ] the underlying Lucene exception of type AlreadyClosedException is a subtype of IllegalStateException, this is more of a 409, if the exception wouldn't have been turned into QueryShardException, it would have been treated as internal server error. In both cases transform would have retried.
  • [ ] The Lucene exception lacks useful information, e.g. the name of the analyzer (not sure if this is possible)
  • [ ] SearchExecutionContext should add important information to the exception, if possible, e.g. name of the field, the sub-query, ...
  • [ ] Transform uses getDetailedMessage() to populate the reason field in _stats to persist the error(_stats is used in the diagnostics bundle). The reason field lacks useful information. Here are some requirements, proper naming, API design, UI implications, to be discussed as separate issue:
  • [ ] transform should log/persist the last executed query in order to help debugging, this could be added as additional field (only visible via _stats, no UI)
  • [ ] transform should use the toXContent(...) method of ElasticsearchException together with proper params to populate another field with the full JSON output of this method. In contrast to getDetailedMessage(), this includes the stack trace

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)