Migrate to opensearch python client library

MJedr commented 1 year ago

Since we are using OpenSearch cluster we should use an OpenSearch python client too.

The migration will enable migrating to python 3.10 (#188).

Unfortunately, some of the dependencies should be updated: for hep these are:

pytest-invenio -> update from 1.2.1 to >2
Invent-indexer -> update from 1.1.1 to >2.1.0
invenio-oaiserver -> update from 1.1.1 to >2.1.0
invenio-search -> update from 1.2.3 to 2.1.0
inspire-utils -> migrate to opensearch v2 and bump in  hepcrawl, inspire-dojson, inspire-json-merger, inspire-matcher, inspire-query-parser, refextract
Invenio-records-rest -> update invenio-search

For next it's more complex since we can't bump invenio packages to the recent versions cause they do not support python 2

drjova commented 1 year ago

We have to identify the following:

[ ] Identify which tasks are accessing directly the ES indices
[ ] Check if opensearch SDK 2 is compatible with the ES 7.10
[ ] Check if elasticsearch SDK is compatible with OpenSearch cluster

MJedr commented 1 year ago

Check if opensearch SDK 2 is compatible with the ES 7.10 -> yes Tasks on next using ES (currently in use)

normalize_collaborations
normalize_author_affiliations
exact_match, fuzzy_match
run_next_if_necessary
holdingpen_search_factory

However, we use ES in other modules (as orcid, records api, etc).

Check if elasticsearch SDK is compatible with OpenSearch cluster -> hard to say, according to the Opensearch docs: Note that while the OpenSearch API is backwards compatible, some clients or tools may include code, such as version checks, that may cause the client or tool to not work with OpenSearch. and Most clients that work with Elasticsearch OSS 7.10.2 should work with OpenSearch, but the latest versions of those clients might include license or version checks that artificially break compatibility. This page includes recommendations around which versions of those clients to use for best compatibility with OpenSearch. Opensearch documentation recommends using python client in version 7.13.4

cern-sis / issues-inspire

Migrate to opensearch python client library #256