Missing folder `plugins/opensearch-knn/lib`

[AlexRuiz7]

Description

Related issue: #99

The plugins/opensearch-knn/lib and all its contents are missing in our packages. From Wazuh, we do not use this plugin for any feature, yet it is included on wazuh-indexer distributions. We need to check if these files are present on upstream, and if so, add them back.

Check Shared libraries section on #99.

Tasks

• Check if these files are present in OpenSearch distributions.
• Study where are these files stored (opensearch-build or plugin's repo)
• During packages assemble, download and add these files to their location.

[AlexRuiz7]

Using OpenSearch's Docker compose, I tried to remove and reinstall the opensearch-knn plugin to check whether the lib folder is installed.

OpenSearch version:

[opensearch@ae29f5538322 ~]$ bin/opensearch --version
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.11.1.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
Version: 2.11.1, Build: tar/6b1986e964d440be9137eba1413015c31c5a7752/2023-11-29T21:43:10.135035992Z, JVM: 17.0.8

opensearch-knn plugin removal:

[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-security
opensearch-security-analytics
opensearch-sql
[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin remove opensearch-knn
ERROR: plugin [opensearch-knn] cannot be removed because it is extended by other plugins: [opensearch-neural-search]
[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin remove opensearch-neural-search
-> removing [opensearch-neural-search]...
[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin remove opensearch-knn
-> removing [opensearch-knn]...
[opensearch@ae29f5538322 ~]$ ls plugins/opensearch-knn
ls: cannot access 'plugins/opensearch-knn': No such file or directory

opensearch-knn plugin installation:

[opensearch@ae29f5538322 ~]$ bin/opensearch-plugin install --batch --verbose org.opensearch.plugin:opensearch-knn:2.11.1.0
-> Installing org.opensearch.plugin:opensearch-knn:2.11.1.0
Checking if url exists: https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-knn/2.11.1.0/opensearch-knn-linux-x64-2.11.1.0.zip
-> Downloading org.opensearch.plugin:opensearch-knn:2.11.1.0 from maven central
Retrieving zip from https://repo1.maven.org/maven2/org/opensearch/plugin/opensearch-knn/2.11.1.0/opensearch-knn-2.11.1.0.zip
- Plugin information:
Name: opensearch-knn
Description: OpenSearch k-NN plugin
Version: 2.11.1.0
OpenSearch Version: 2.11.1
Java Version: 11
Native Controller: false
Extended Plugins: [lang-painless]
 * Classname: org.opensearch.knn.plugin.KNNPlugin
Folder name: 
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission loadLibrary.opensearchknn_faiss
* java.lang.RuntimePermission loadLibrary.opensearchknn_nmslib
* java.net.SocketPermission * connect,resolve
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.
-> Installed opensearch-knn with folder name opensearch-knn

lib folder is not present:

[opensearch@ae29f5538322 ~]$ ls plugins/opensearch-knn/
LICENSE.txt  commons-lang-2.6.jar     guava-32.0.1-jre.jar         plugin-descriptor.properties
NOTICE.txt   failureaccess-1.0.1.jar  opensearch-knn-2.11.1.0.jar  plugin-security.policy

[AlexRuiz7]

Issue created in OpenSearch:

• https://github.com/opensearch-project/k-NN/issues/1408
• https://github.com/opensearch-project/k-NN/issues/2197

[AlexRuiz7]

OpenSearch's developers replied and as a summary, these libraries are platform dependent. I couldn't find these files anywhere, so these are probably included during building in Jenkins. The complete plugin can be downloaded from their CI system. Each distribution package includes a manifest.yml file with the URL of each component, looking like this:

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.8.0/7935/linux/x64/tar/builds/opensearch/plugins/opensearch-knn-2.8.0.0.zip

We could easily install this zip during the packages' generation, as we the link is parametrized as follows:

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/$VERSION/$CI_RUN_ID/$PLATFORM/$DISTRIBUTION/$ARCHITECTURE/builds/opensearch/plugins/opensearch-knn-$VERSION.0.zip

However, we miss the $CI_RUN_ID, which can only be found in the manifest.yml file itself.

For our use case, this means that we either download the whole bundle and read the link from its manifest.yml, or simply copy the libs/ folder from the bundle to our package. Both options would require downloading a distribution of OpenSearch, which we want to avoid at all costs. Distribution links look like this:

https://artifacts.opensearch.org/releases/bundle/opensearch/2.11.1/opensearch-2.11.1-linux-x64.tar.gz

A better approach would be to add an environment variable on GH Actions with the download link or the CI_RUN_ID number of the OpenSearch version that wazuh-indexer is based on. We would need, however, to create and update this variable manually each time we upgrade to a newer upstream version. Our assemble.sh script would need to be updated to manage this special case.

An automatization of this process would be to download an OpenSearch package, extract it, read the link from the manifest.yml file and create/update the environment variable on GH Action if it's unset or outdated, so this step isn't executed on every workflow run. In this case, in order to determine whether the variable is outdated, we would need to use the full download link, so we can compare the versions.

We are yet to discuss if we should bundle this package as part of wazuh-indexer, so the issue stays on Triage until then.