search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.57k stars 2.79k forks source link

[QUESTION] Possible to deploy multiple models using ManagedOnlineDeployment ? #35435

Closed matthiasschuurmans closed 3 months ago

matthiasschuurmans commented 5 months ago

Dear Microsoft,

I've recently dug into the azure ML studio platform, including its ability to deploy models as endpoints. My use case could potentially involve many models that get called maybe a few times per day, so I'm interested in running many models on a single VM to reduce costs, and I'm not worried about overloading the VM.

I'm not super familiar with all the different deployments, computes, VMs etc., but from what I understand, deploying multiple models in a single deployment and to a single endpoint (and then having the scoring script figure out what model to use based on the request) is only possible in the SDK v1, not in SDK v2. Is this true?

I found a lot of documentation concerning SDK v2, i.e https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints?view=azureml-api-2&tabs=python , where it seems ManagedOnlineDeployment contains the model to be deployed. In v1, the azure.ml.core.model.Model.deploy() function can deploy a list of models, but ManagedOnlineDeployment seems to only allow a single model. I think I read somewhere that multiple models is supported via CLI v2, but I prefer deploying with a Python script due to some technicalities in the deployment procedure.

I also ask, because I found out that an endpoints deployed with v1 is not listed when I then try to list it via v2 (azure.ai.ml.MLClient.online_endpoints.list(). So by deploying with v1, I also seem to reduce myself to approaching them via v1. Is this true?

I cannot seem to find the links to the Microsoft documentation anymore, I would appreciate any links that prove/disprove my assumptions.

github-actions[bot] commented 5 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

pvaneck commented 5 months ago

Thanks for the feedback. Can someone from @azureml-github provide insight here?

matthiasschuurmans commented 4 months ago

Hey, any followup here would be appreciated.

Jyankc commented 4 months ago

Hi , Help with this is needed. Can you guide us on how to upload multiple models in sdk v2?

bricew25 commented 4 months ago

Hello, I am also looking for more documentation on this.

Montalvovich commented 3 months ago

I've been wondering the same thing. We've been trying to upload multiple models into a single deployment in Azure, but it appears we only have the option to upload a single model at a time. I'd love more guidance to this or documentation. It looks like it may have been possible previously, but the documentation was deleted out of GitHub. Thanks in advance.

bricew25 commented 3 months ago

Dear Microsoft,

I've recently dug into the azure ML studio platform, including its ability to deploy models as endpoints. My use case could potentially involve many models that get called maybe a few times per day, so I'm interested in running many models on a single VM to reduce costs, and I'm not worried about overloading the VM.

I'm not super familiar with all the different deployments, computes, VMs etc., but from what I understand, deploying multiple models in a single deployment and to a single endpoint (and then having the scoring script figure out what model to use based on the request) is only possible in the SDK v1, not in SDK v2. Is this true?

I found a lot of documentation concerning SDK v2, i.e https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints?view=azureml-api-2&tabs=python , where it seems ManagedOnlineDeployment contains the model to be deployed. In v1, the azure.ml.core.model.Model.deploy() function can deploy a list of models, but ManagedOnlineDeployment seems to only allow a single model. I think I read somewhere that multiple models is supported via CLI v2, but I prefer deploying with a Python script due to some technicalities in the deployment procedure.

I also ask, because I found out that an endpoints deployed with v1 is not listed when I then try to list it via v2 (azure.ai.ml.MLClient.online_endpoints.list(). So by deploying with v1, I also seem to reduce myself to approaching them via v1. Is this true?

I cannot seem to find the links to the Microsoft documentation anymore, I would appreciate any links that prove/disprove my assumptions.

Im not sure how much this helps, but when we specify the folder instead of the files, it looks like it will upload all the models in that folder. Similar to in the GUI, you can choose file vs folder upload. model = Model(path="./model/")

After running the build with SDKv2 we see multiple artifacts when the model is deployed.

PratibhaShrivastav18 commented 3 months ago

@matthiasschuurmans @Jyankc @bricew25 @Montalvovich please follow this sample notebook to deploy multiple models using ManagedOnlineDeployment via SDK v2.

matthiasschuurmans commented 3 months ago

Thank you @PratibhaShrivastav18 . Is that notebook also linked to and explained somewhere in the documentation? If yes, can you please link it, and if no, would it be possible to add?

PratibhaShrivastav18 commented 3 months ago

@matthiasschuurmans No, this notebook is not available on learn.microsoft.com, however you can use this article that links to other multiple model samples and we can consider adding this one also in the future.