search

kserve / modelmesh-serving

Controller for ModelMesh
Apache License 2.0
203 stars 114 forks source link

Need support on manipulating model to pod in order to deploy ensemble model #313

Open vulong3896 opened 1 year ago

vulong3896 commented 1 year ago

Describe your proposed solution

I am trying to deploy ensemble model to Triton with ModelMesh. In order to do this I need a mechanism to ensure that all the sub models have to be in a same pod. Currently, ModelMesh randomly segrerate my models into different pods so the ensemble will be failed. Do you have any plan for this in the future? If not please guide me where I can modify the source code in order to achieve this!

njhill commented 1 year ago

@vulong3896 thanks for your question. This requirement has come up before, and I'd given it quite a bit of thought some time ago (prior to model-mesh being open sourced). I've copied notes from an internal issue to a new one here: https://github.com/kserve/modelmesh-serving/issues/317.

Do you have any plan for this in the future?

No plans at this moment but if you are interested in taking it on I can help to guide/review!

vulong3896 commented 1 year ago

@njhill thanks for your quick reply

Kokkini commented 1 year ago

@njhill My current project also needs this feature. If you have no plan for this yet, I can try implementing it. Could you give me some pointers on where to start and an estimate of the effort it would take?