[PROPOSAL] Make the Opensearch LTR Plugin a part of the OpenSearch official plugin ecosystem

[gsingers]

What/Why

What are you proposing?

A common approach to improving ranking and relevance today is via machine learning and specifically learning to rank approaches. We wish to propose the incorporation of the community build OpenSearch LTR plugin into the official OpenSearch ecosystem. Derived from the ElasticSearch Learning to Rank (LTR) plugin (which was derived from the Solr plugin), the OpenSearch LTR plugin was originally ported by Alberto Paro (original) and then forked and maintained by Grant Ingersoll as a fork on Github. It consists of code, tests and Docker build of OpenSearch with the plugin installed.

What users have asked for this feature?

LTR is a well-documented feature of most modern search engines. Daniel Tunkelang and I teach a class on search with machine learning that uses this plugin. Based on it's use in Elasticsearch, I suspect there are many others who would benefit from it, but I have not done extensive research on that.

What problems are you trying to solve?

Provide higher quality ranking and relevance through the use of machine learning.

What is the developer experience going to be?

While not 100% the same, you can get a general idea via the community LTR plugin for Elasticsearch maintained by OpenSource Connections.

Are there any security considerations?

None that I am aware of.

Are there any breaking changes to the API

None that I am aware of.

What is the user experience going to be?

See the docs: https://elasticsearch-learning-to-rank.readthedocs.io/en/latest/

Are there breaking changes to the User Experience?

No

Why should it be built? Any reason not to?

It's already built, this proposal would bring it under the OpenSearch umbrella.

What will it take to execute?

Primarily bringing over the code and hooking it into the build and deployment system.

Any remaining open questions?

ideas for future:

• work more closely with other ML capabilities
• Expand the number of models and approaches supported
• Improve the docs
• Add support for additional data features

[macohen]

Thanks, @gsingers. We should let this collect some feedback from the community and maintainers of OpenSearch. This looks like a good starting point to me for an open source plugin.

CC: @dblock, @elfisher, @sean-zheng-amazon, @macrakis, @epugh, @vamshin, @msfroh

[CarlMeadows]

+1

We run a version of this plugin on Amazon OpenSearch Service - see here. However, we have not advanced the plugin since we forked it to make it run on OpenSearch. I would love to see continued development of the original LTR plugin for OpenSearch users and making it part of the project would make that easier. It would also make leveraging those enhancements for Amazon OpenSearch Service users much more straightforward.

[elfisher]

I really like this proposal! We would probably want to add documentation under the opensearch.org doc site as part of this too.

[nateynateynate]

This is a feature that can have a lasting impact on just about every actual use case I can think of. Let's get it in here and treat it well.

[nknize]

I'm 💯 supportive of this and will help get it pulled in. Strong history to support this request. Thanks @gsingers!

[dblock]

💯 let us know how we can help @macohen

[epugh]

One lesson from the ES version of the LTR plugin is that most of the work that happens on that plugin was around staying current with Elasticsearch, and only a very small amount of work went into improving the suitability of the plugin for supporting LTR use cases. Just looking at @gsingers commit log, I unfortunately see that same pattern unfolding for the OS version (which is probably what prompted this proposal!). Assuming that bringing the plugin into the official ecosystem solves that problem, then I see a big upside for LTR in OpenSearch.

I hope moving it into the official ecosystem opens the door to MORE innovation around how ML can work with OpenSearch, and that the plugin itself starts to evolve quicker.

[dblock]

Thanks @epugh!

In the short term adding the plugin into the default distribution would indeed alleviate many of these problems as we have automated our way through it - for example when an upstream interface changes and breaks downstream plugins, we auto-cut GitHub issues.

If you have any interest, check out https://github.com/opensearch-project/opensearch-sdk-java. We will launch this as an experimental feature in 2.8.0. It decouples the rigid dependency and you can write ~a plugin~ an extension that works with a range of versions of OpenSearch, which means no more maintenance in staying current! For now there are some out-of-proc downsides, but we will bridge those over time.

[macohen]

@gsingers, @aparo, @epugh I'm thinking that we bring in Grant's repo into the project and there is no upstream. It looks like the original fork by Alberto is on hold, at least for now. Given the future of OpenSearch with extensions we may consider factoring out a core LTR library for use across engines in the future. For now, would it make sense to move the repo under opensearch-project (as opposed to forking it) and investigate the ES LTR plugin for parity later?

[gsingers]

@macohen Yes, I think let's move the repo under and then we can bring to parity (and add to) later.

[macohen]

Great. We're looking forward to working with you on this Grant. I'll work with you and our Open Source Program Office to facilitate the move and leave this open until we're done with the move.

CC: @hyandell

[mattsb42-aws]

Hi @gsingers. I see that you invited me to the repo, but only repo owners (the owning user, in the case of a user-account-owned repo) can transfer ownership of repos. Please transfer ownership to my account and I can move transfer it into this org.

[gsingers]

@mattsb42-aws OK, sent. Sorry for the delay.

[mattsb42-aws]

Thanks, @gsingers; I'll get things moving along.

[macohen]

The repo is here now: https://github.com/opensearch-project/opensearch-learning-to-rank-base. Thanks @mattsb42-aws. I also created a PR to set the ADMIN.md, MAINTAINER.md and CODEOWNERS up properly to start. https://github.com/opensearch-project/opensearch-learning-to-rank-base/pull/7. We could consider renaming the repo as well. @gsingers, if you're satisfied, would you be open to creating specific issues and PRs in the moved repo and closing this one out?

[dblock]

This part is done, so I'm making an executive decision to close it ;)