Investigate `ClusterStorageContainer` for ModelMesh deployments

[lampajr]

Based on https://github.com/opendatahub-io/model-registry/issues/254 initial findings it seems the new ClusterStorageContainer feature is not supported in ModelMesh.

Explore the possibility to make a similar feature working on ModelMesh deployments as well, i.e., unlock the possibility to manage custom URI storage locations like model-registry://....

[rareddy]

Is there a corresponding issue on the upstream model mesh? I would think there needs to be some service provider uri boot stapping that needs to be done in a core component.

[lampajr]

Is there a corresponding issue on the upstream model mesh? I would think there needs to be some service provider uri boot stapping that needs to be done in a core component.

+1 it should require some architectural rethinking from modelmesh point of view to reuse a similar feature because the actual modelmesh implementation does not make use of initContainers to fetch/download the model. I created an issue to get some more information and to keep track of it: https://github.com/kserve/modelmesh-serving/issues/471 but no updates so far from modelmesh side

[lampajr]

fwiw I am currently exploring if we can, at least, have some ad hoc changes on our side. i.e., in [opendatahub-io/modelmesh-serving] for instance somewhere here https://github.com/opendatahub-io/modelmesh-serving/blob/ec13e8d296aa6d67d89365ca26cab3acc85d3f3a/pkg/predictor_source/inferenceservice_registry.go#L164-L198 - still need to evaluate, just some initial ideas