Open dbwiddis opened 1 month ago
The location of this code in ML-Commons is temporary; it will eventually be migrated somewhere else (possibly a module on the core project, possibly a separate plugin, possibly the Java SDK project).
Do we have any ETA on getting the final decision on this?
Expect PRs and Tasks to accomplish the following:
Do we have any order to follow? Like may be performing sanity testing first on the branch? Also integration tests are missing here?
The location of this code in ML-Commons is temporary; it will eventually be migrated somewhere else (possibly a module on the core project, possibly a separate plugin, possibly the Java SDK project).
Do we have any ETA on getting the final decision on this?
No. I've mentioned the need to decide on this "sooner rather than later" but it hasn't seemed to prompt action.
Is there a particular deadline you think is needed that we can push toward? Should we delay migrating from the feature branch until it's decided? @andrross do you have any input here?
Do we have any order to follow?
My primary goal was articulating the tasks first.
The default implementations can probably follow the sequence outlined; most testing already exists for those, and I expect we might only find a few misses.
The alternate data store implementations can / should proceed in parallel and may require a lot of new tests to be written.
Like may be performing sanity testing first on the branch?
I'd appreciate your input here as a maintainer on what you require to merge this.
It sounds like we should keep developing on the branch until we're satisfied? I thought there was a push to get this on main.
Also integration tests are missing here?
I've listed integ tests 3 times above. Is there a specific case I'm missing? To elaborate:
+1 on reusing existing integ tests to test against remote opensearch and DDB stores.
From an ordering perspective, we should add new integ tests in feature branch before merging them to main.
From an ordering perspective, we should add new integ tests in feature branch before merging them to main.
+1
The location of this code in ML-Commons is temporary; it will eventually be migrated somewhere else (possibly a module on the core project, possibly a separate plugin, possibly the Java SDK project).
Do we have any ETA on getting the final decision on this?
No. I've mentioned the need to decide on this "sooner rather than later" but it hasn't seemed to prompt action.
Is there a particular deadline you think is needed that we can push toward? Should we delay migrating from the feature branch until it's decided? @andrross do you have any input here?
My thinking here is that there is a potential project to introduce a proper metadata abstraction into the core (for use by the core itself and plugins) as described here. The interfaces you've defined in ml-commons may well be part of that project, but I think we'd need at least a design for this work before we'd know that. Given that plugins want to move forward with this work ahead of doing the core work, I'd lean towards putting these interfaces in a library outside of the core in the short term since the core itself will not be using them.
I will be starting the process of migrating the code developed on the
feature/multi_tenancy
branch tomain
. I'm creating this issue to establish context for reviewers and pre-answer some frequently asked questions.Important context to understand:
There are two main purposes of this branch/feature:
There will be new packages created in the
opensearch-ml-common
module:,org.opensearch.sdk
containing a new metadata interface and associated request/response objects, andorg.opensearch.sdk.client
containing a default client implementation based on NodeClient, which implements existing behavior.opensearch-ml-plugin
moduleorg.opensearch.ml.sdkclient
package. Like the above interfaces, this is not their final location.Because of the way the
createComponents()
method on thePlugin
interface works (binding those instances to their runtime class, not their interface), the client will be a concrete class which delegates method calls to an internal delegate instance. This is not the ideal object-oriented structure, but seemed the best approach to simplify injection of the client into classes which are instantiated insidecreateComponents()
and would otherwise not have access to an interface-bound instance created in a separate module.The vast majority of changes across the codebase are copy-and-paste using the following general format (Get is shown, similar for Index, Update, Delete, and Search):
Old code (some variations on wrappers or anonymous subclasses, etc.):
New code (some variations):
Future
rather than anActionListener
, and the multi-threaded execution.Current test:
Updated test to account for multithreading and client args
Expect PRs and Tasks to accomplish the following:
Add any questions/comments!