kitodo / kitodo-production

Kitodo.Production is a workflow management tool for mass digitization and is part of the Kitodo Digital Library Suite.
http://www.kitodo.org/software/kitodoproduction/
GNU General Public License v3.0
64 stars 63 forks source link

Add support for OpenSearch #6131

Closed stweil closed 1 week ago

stweil commented 4 months ago

This PR adds support for OpenSearch while still working with Elasticsearch.

It is for people who want to work with recent code, newer OpenJDK versions (17, 22, but 11 also still supported) and OpenSearch because they cannot use Elasticsearch.

It's also for people who want to use Elasticsearch with newer code, either for security reasons or because they want to work with newer OpenJDK versions.

stweil commented 4 months ago

The latest code can be built with up to OpenJDK 22.

solth commented 4 months ago

@stweil thanks for sharing this development. It is good to see that compatibilty with OpenSearch can be achieved this way. However, the current plan for Kitodo.Production is not to switch from ElasticSearch to the open source fork OpenSearch, but rather to incorporate the HibernateSearch framework to allow institutions to decide which search server variant they want to use (see #5760 and #6065 for details).

Of course your development is of value anyway, but I do not think creating a pull request with these changes is the right way of giving users access to it and it might trigger wrong expections (even if its just a draft), because - as you said - it won't be merged. As mentioned above, replacing the ElasticSearch client libraries in Kitodo.Production with corresponding OpenSearch libraries is currently not on the roadmap. Instead I would suggest to link the branch that this pull request is based (stweil:opensearch) in the discussion and issue mentioned above so that users interested in trying it out can access it that way.

stweil commented 2 months ago

I rebased and minimized the pull request now. Based on PR #6215 there remains a single small commit.

With this pull request it is directly possible to use Elasticsearch as before (see CI).

The PR replaces old Elasticsearch module by newer OpenSearch modules, so OpenSearch can also be used instead of Elasticsearch (tested locally with the same commands as in CI test, but an OpenSearch server).

It's also possible to rebase the existing work for HibernateSearch on top of this PR with only a few merge conflicts which are easy to fix.

solth commented 1 month ago

@stweil I just tested your branch and can confirm that it works direktly "out-of-the-box" with my existing ElasticSearch-Server, without the need for any other adjustments. Additionally, it seems all changes in the code - apart from the new dependencies in the pom files - amount to just the elasticsearch import statements being replaced with corresponding opensearch imports, withtout any changes to the usage in the actual business logic at all. I have to admit I am suprissed that facilitating OpenSearch compatibility while maintaining ElasticSearch compatibility was possible with so few changes.

That said, I do not see the advantage of having these changes in the code when they are replaced by HibernateSearch before the next larger release, e.g. 3.8. Therefore I think merging this pull request would only make sense if we shifted the HibernateSearch integration to a new release 3.9 and released a version 3.8 of Kitodo in the near future with the OpenSearch compatibility and the new functions for the metadata editor for which pull requests have already been opened.

Since the HibernateSearch integration has the highest priority right now, I wouldn't want to introduce any changes to the code that would impede that project, so I would like @matthias-ronge decide whether he agrees with these changes or whether he thinks they would further postpone the finalization of the HibernateSearch integration for him.

stweil commented 1 month ago

Recent changes (0e43651ad5951dbcf3859d3b1fc83953e86a8eb8, 1f6a523d69c7cee14654b41646a950c084bd0c9d) caused a merge conflict which is now fixed.

stweil commented 2 weeks ago

I had to rebase the PR and fix a merge conflict caused by commit 8f2015b848d4f2a1b928b652cf72adf9fd8d43e7.

It's now again ready for merging.

stweil commented 2 weeks ago

Now rebased and fixed new merge conflict.

stweil commented 1 week ago

Rebased and fixed merge conflict.