[ESQL] Aggregations - return null values for individual buckets on errors

not-napoleon commented 2 months ago

ESQL's design philosophy has been to avoid failing queries on errors as much as possible. Scalars achieve this by trapping (expected) errors and setting their value to null for that row, with a warning. Aggregations, however, do not currently do this. Supporting null + warning for aggs has several steps:

[ ] Make Source and Warnings objects accessible from within the generated aggregations code
[ ] Extend the @Aggregator annotation to include a list of exceptions that should be turned into warnings, similar to warnExceptions on the @Evaluator annotation, and update the code generation to catch those warnings in the appropriate place
[ ] Provide some means for aggregations to mark buckets as null. This probably looks like a bitset column.

That last step will require a change to the Layout. We currently have no capacity for evolving the Layout, which must be the same for all nodes participating in the query. Solving that will likely be the bulk of the work in this ticket.

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-analytical-engine (Team:Analytics)

yeswanth-koti26 commented 2 months ago

Hello @elasticsearchmachine Elasticsearch team,

I am interested in contributing to the issue regarding the handling of errors during ESQL aggregations by returning null values for individual buckets. I've read through the discussion and I understand the objective of making aggregations more resilient to errors. I would like to work on this and would appreciate any guidance or suggestions you might have to get started. Thank you!

Best regards, @yeswanth-koti26

not-napoleon commented 2 months ago

Hello @yeswanth-koti26 and thanks for your interest in contributing to Elasticsearch. This particualr issue is not really ready to be worked. There are a lot of hidden complexities around how we will need to evolve the page layout negotiation in a backwards compatible way. Finalizing a design for that is something we need to do within the ES|QL team. If you're looking for a good place to get started contributing, I recommend picking a function off of this list to implement. Please also make sure you've read our contributing guidelines, especially the part towards the end about contributing as part of a class.

elastic / elasticsearch

[ESQL] Aggregations - return null values for individual buckets on errors #110443