search

opensearch-project / custom-codecs

OpenSearch custom lucene codecs for providing different on-disk index encoding (e.g., compression).
Apache License 2.0
6 stars 15 forks source link

[Documentation] communicate the technical merits and tradeoffs of using ZStd Compression Codec #108

Open nknize opened 11 months ago

nknize commented 11 months ago

This issue is created out of opensearch-project/OpenSearch#9422. Technical discussions around the merits and tradeoffs of users choosing ZStd codec have been discussed. We need to make sure this doesn't get lost in the noise of how to react to the potential demotion of the ZStd module to a plugin. I'm opening this issue here to capture those merits and tradeoffs until we can get them into proper documentation.

ZStd Pros:

  1. better / faster compression
  2. cost savings

Cons:

  1. force killed nodes
  2. wildcard loadLibrary security hole
  3. no rolling upgrade
  4. libc portability issues

This is a "living" issue and the list will likely be updated as more information is learned and new protection mechanisms are injected.

msfroh commented 6 months ago

@opensearch-project/admin, can we please move this to the https://github.com/opensearch-project/custom-codecs repo?

dblock commented 2 months ago

[Triage -- attendees 1, 2, 3, 4, 5, 6, 7]