search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.56k stars 24.35k forks source link

Fork HdrHistogram library #95904

Open martijnvg opened 1 year ago

martijnvg commented 1 year ago

We plan to fork the HdrHistogram library.

There are two main reasons behind this choice:

  1. We would like to control semantic version and backward compatibility according to our definition. Right now, for instance, TDigest does not match our usage of semantic versioning when changing the library code and that makes upgrading quite challenging because exposes us to backward compatibility issues.
  2. We would like to change those libraries to use some specific Elasticsearch libraries/tools/frameworks such as BigArrays. Right now when running some aggregations (percentiles, boxplot,...) we experience OOMs due to large memory usage. Using BigArrays, for instance, would allow us to deal with OOMs using Circuit Breakers.

The immediate goal is just to fork and then later enhance the forked library to make use of big arrays infrastructure.

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-analytics-geo (Team:Analytics)

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-delivery (Team:Delivery)

mark-vieira commented 9 months ago

Closed by https://github.com/elastic/elasticsearch/issues/95903.

martijnvg commented 9 months ago

This hasn't been completed yet. Only t-digest library has been forked, but not yet the HDR histogram library.

mark-vieira commented 9 months ago

This hasn't been completed yet. Only t-digest library has been forked, but not yet the HDR histogram library.

Sorry about that, I misread the discussion as being we weren't going to fork this and only t-digest.

elasticsearchmachine commented 4 days ago

Pinging @elastic/es-analytical-engine (Team:Analytics)