[FEATURE] Add support for experimental codecs

opensearch-project / custom-codecs

OpenSearch custom lucene codecs for providing different on-disk index encoding (e.g., compression).

Apache License 2.0

6 stars 13 forks source link

[FEATURE] Add support for experimental codecs #148

Open mgodwan opened 1 month ago

mgodwan commented 1 month ago

Is your feature request related to a problem?

Coming from https://github.com/opensearch-project/custom-codecs/pull/122#issuecomment-2109681404

Given custom-codecs plugin is out of sandbox, any new codecs introduced should get some bake time to be tried by users before it can be marked ready for production usage. We should add some setting to allow-list a codec to be generally available. Codecs like zstd have been existing for quite a few versions, and already have seen backward compatibility handled and working. As new codecs are added (e.g. qat based), it would be helpful to enable them for production usage only after they have been vetted through some community feedback as well.

What solution would you like?

Introduce a setting which allows a codec to be marked experimental, and enabled on demand.

What alternatives have you considered?

Introduce codecs in new sandboxed plugins

reta commented 3 weeks ago

Coming from https://github.com/opensearch-project/OpenSearch/pull/13992#issuecomment-2160785991, I think we have an agreement to:

we want to limit enable / disable controls to the custom-codecs plugin only
we want to have enable / disable controls on individual codecs level

We have separate sources of codecs:

Codecs SPI (that come from Apache Lucene)
CodecService that manages codecs on OS side

The CodecService is provided by custom-codecs and it is easy to implement enable / disable controls on individual codecs level (using plugin specific settings fe). On the other side, Codecs SPI of Apache Lucene does not support any kind of codecs filtering but its usage (and impact) is somewhat limited (to be confirmed during implementation).

sarthakaggarwal97 commented 2 weeks ago

@reta I think that's a good summary. Do you have an approach in mind on how to restrict the changes to custom-codecs?

reta commented 1 week ago

@reta I think that's a good summary. Do you have an approach in mind on how to restrict the changes to custom-codecs?

@sarthakaggarwal97 sorry for delay, I didn't forget about it but there are quite related discussions happening around the same mechanism but different subjects (https://github.com/opensearch-project/OpenSearch/pull/14479), will suggest the solution once we settle on the approach.

dblock commented 1 week ago

Catch All Triage - 1 2 3 4 5 6