search

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Apache License 2.0
1.87k stars 175 forks source link

Add "deploy.py" file for deployment in AzureML into documentation #184

Open natasha-savic-msft opened 1 year ago

natasha-savic-msft commented 1 year ago

Hello, I followed you steps for DeepSpeed deployment. Could you kindly add another step between setting up the az ml CLI and deploying the mii resource. To achieve this, create a new file called e.g. deploy.py run python -m deploy. Then proceed with the steps from cd ./bloom560m-deployment_aml. It is not clear currently that a new file is needed to perform the deployment containing the code snippet under "Deployment"

image
natasha-savic-msft commented 1 year ago

Since no requirements.txt is provided for AzureML deployment, please also add dependencies. E.g. the transformers library needs an upgrade (huggingface-hub-0.14.1 transformers-4.29.2) otherwise deploy.sh fails with:

Traceback (most recent call last): File "model_download.py", line 19, in <module> _ = AutoConfig.from_pretrained(model) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 632, in from_pretrained config_class = CONFIG_MAPPING[config_dict["model_type"]] File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py", line 347, in __getitem__ raise KeyError(key) KeyError: 'bloom'