[P0] [SPIKE] Remote model deployment

[heyselbi]

From req doc: Remote deployment (eg. locations other than the cluster where model deployment is initiated). Note: this does not include edge locations

• Support models being deployed to remote (location other than where model deployment is initiated):
  • P0: OCP including single node OCP
  • This is the standard OCP product. Other variants such as MicroShift will be covered in a separate edge epic

Acceptance criteria:

• [ ] Architecture or detailed explanation of what remote deployment entails
• [ ] Scope of the work that would be needed (perhaps in L0, L1 etc format)

[israel-hdez]

There are two proposals that have been written over time:

1. @anishasthana proposal: GDoc link
   • It is the older one, yet still quite valid.
   • It mentions a way to achieve remote deployment right now by using ACM/OCM and its GitOps/ArgoCD integration. This way is somewhat manual, although it requires no changes to ODH.
   • It proposes a new architecture to make it easier for the user to deploy models remotely from a hub cluster. This would, again, rely on ACM/OCM but management would be done through ODH using 3 new CRDs rather than via a GitOps repository.
2. @israel-hdez proposal: GDoc link
   • It is the newer one. Despite this, the proposed architecture is almost identical to the one from @anishasthana.
   • It describes slightly deeper the technical details that would need to be worked to implement remote model deployment.
   • In general, it is complementary to the proposal from @anishasthana .

Both documents are focused on ModelMesh while, nowadays, ODH has also adopted KServe. However, since ModelMesh and KServe share the same APIs, both proposals can be extended to KServe without changes to their essence.

There is also a related and ongoing work focused on deploying models "at the edge" that can be seen in this repository: https://github.com/opendatahub-io/ai-edge/. Currently, the ai-edge work is closer to @anishasthana "current state" ("right now" way) about using ACM/OCM and its GitOps/ArgoCD integration as helpers for deploying models. In the ai-edge work a pattern using existing tooling is being proposed meaning that most of the setup is still on user side.

In general, the "remote model deployment" requirement is too general and does not make it clear how much 3rd party tooling is OK to use, nor how much integration to ODH is needed that would simplify/reduce manual intervention. This needs to be refined.

That said, an OCM/ACM <-> ODH integration may still be the solution that makes most sense to provide remote model deployment capabilities into ODH:

• At the very least, ACM/OCM should be used because of the cluster registration capabilities.
• Secondarily, it makes sense to use ACM/OCM's multicluster-observability-operator to collect metrics into a central location.
• Any other technical gaps would be solved through an ODH-OCM/ACM add-on.
  • Additional APIs/CRDs would need to be created to cover any technical gaps.
  • The complexity of the add-on would be dependent on how much capabilities we want to cover into ODH in favor of relying on other existent tooling.

[israel-hdez]

Assuming ACM/OCM setup and cluster registration would be out of scope for model-serving (i.e. the user would do it through ACM/OCM) and also that setup and deployment of the multicluster-observability-operator would be out of scope, the work to implement remote deployment would be limited to creating an ACM/OCM add-on that would provide the missing capabilities.

Breakdown of tasks

The breakdown of tasks assume that the other clusters would be managed from a hub cluster and that support for direct deployment on the other clusters is out of scope (i.e. creating an InferenceService directly in the other clusters could be possible, but is not going to be possible to manage such ISVC from the hub cluster).

Back-end support

• Scaffolding/creation of a new ACM/OCM add-on for model serving (agent and manager) - it could be part-of an already existent project (e.g. odh-model-controller).
  • Implement deployment, configuration and undeployment of the ModelMesh stack in the other clusters (headless).
  • Implement (or validate) configuration of metrics collection to the "local" OpenShift, assuming the multicluster-observability-operator is going to be already configured to federate metrics to the hub cluster.
  • Implement deployment, configuration and undeployment of the KServe stack in the other clusters (headless).
  • Implement (or validate) configuration of metrics collection (similar to ModelMesh case).
  • Implement enabling/disabling ModelMesh on the other clusters (KServe shouldn't need to be enabled).
  • Implement management of storage credentials in the other clusters.
  • Implement management of ServingRuntimes in the other clusters.
  • Implement management of InferenceServices in the other clusters.
• ODH operator support for deploying the serving ACM/OCM add-on (in case a new project is created).
• Documentation about how to setup/prepare a cluster to work as a hub for model serving (i.e. examples to setup OCM/ACM and register a cluster).
• Documentation about how to setup/prepare the clusters to federate metrics to the hub cluster (i.e. examples of the multicluster-observability-operator).

The previous list enumerates the needed tasks to expose the most important blocks of functionality for model serving. Additional work/tasks would be needed to expose other functionality (like InferenceGraph).

Note: Unless possible with existing APIs, in general the "implement" tasks imply creation of new cluster APIs (CRDs) to manage the resources in the other clusters. For example, to deploy the model serving stack to the other clusters, it would be needed to create a ModelServing CRD to be able to manage the stack from the hub cluster.

Front-end support

Basically, expose functionality from the back-end:

• Ability to deploy or remove the model serving stack on other (already registered) clusters.
• Extend the feature to enable or disable model serving on namespaces/projects to the other clusters.
• Ability to create and remove model servers (ServingRuntimes) on other clusters.
• Ability to configure and remove storage credentials on other clusters.
• Ability to select which cluster (or clusters) to deploy or remove a model.

[israel-hdez]

Closing, as there seems to be no more actionable work.