Lucene99 to backward_codecs per bump to 9.12 in core

opensearch-project / k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data

https://opensearch.org/docs/latest/search-plugins/knn/index/

Apache License 2.0

156 stars 123 forks source link

Lucene99 to backward_codecs per bump to 9.12 in core #2192

Closed finnegancarroll closed 1 month ago

finnegancarroll commented 1 month ago

Description

Lucene bumped in Opensearch here: https://github.com/opensearch-project/OpenSearch/pull/15333 It looks like Lucene99Codec is deprecated and needs to be updated to use the backwards compatible implementation.

https://github.com/opensearch-project/OpenSearch/commit/146b0f7264b1426d475e757729d213097b776212#diff-48f2642fce5a8906ea46c9d4796ed3477e49c6fd91eaaa9ba03f6b840837930bL12

Related Issues

N/A

Check List

[x] New functionality includes testing.
[x] New functionality has been documented.
[x] API changes companion pull request created.
[x] Commits are signed per the DCO using --signoff.
[x] Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

martin-gaievski commented 1 month ago

For the previous lucene codec upgrade the scope is typically much larger then just updating the package name for previous codec version, e.g. https://github.com/opensearch-project/k-NN/pull/1383/files. Are there other changes that are needed for knn in scope of codec upgrade?

finnegancarroll commented 1 month ago

It looks like the above pr both modifies the lucene95 codec to import from import org.apache.lucene.backward_codecs as well as add a KNNCodec implementation for the newly available Lucene99Codec. Here i'm planning to just fix the import for the Lucene99Codec to resolve build issues with the most recent OpenSearch snapshot without adding any new implementation.

finnegancarroll commented 1 month ago

I notice when I build knn the project is not able to fetch the new lucene artifacts for org.opensearch:opensearch:3.0.0-SNAPSHOT.

Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find org.apache.lucene:lucene-core:9.12.0-snapshot-847316d.
     Required by:
         project : > org.opensearch:opensearch:3.0.0-SNAPSHOT
         project : > org.opensearch:opensearch:3.0.0-SNAPSHOT > org.opensearch:opensearch-core:3.0.0-SNAPSHOT

Any idea where these lucene artifacts are usually fetched from?

navneet1v commented 1 month ago

@finnegancarroll we will require more changes for the fix. I am picking up the fix. is that fine with you?

finnegancarroll commented 1 month ago

Absolutely please do. Thanks @navneet1v!