search

machine-learning-apps / ml-template-azure

Template for getting started with automated ML Ops on Azure Machine Learning
MIT License
126 stars 87 forks source link

Unexpected error in aml-deploy action step #13

Closed an185144 closed 3 years ago

an185144 commented 3 years ago

Hi - we are experiencing an unexpected error which we haven't been able to solve for a while now. All the steps before aml-deploy run just fine, but aml-deploy is suddenly not being able to pull the registered model. The name and version of the model coming from aml-registermodel have been cross-checked and are being correctly passed to aml-deploy. Based on the logs looks like there are also no issues with workspace authorization.

Hoping you've seen anything similar happen with the template and can help us.

Attaching the action log, deploy.json parameters and aml_arm_config json. We have kept the workflow largely the same except for adding some debugging actions.

15_Deploy model.txt

deploy.json

{ "name": "test", "deployment_compute_target": "akscompute" }

aml_arm_config from aml-workspace (found in $GITHUB_WORKSPACE/.azureml/)

{"Id": null, "Scope": "/subscriptions/{subscriptionid}/resourceGroups/{resourcegroup}/providers/Microsoft.MachineLearningServices/workspaces/{workspacename}"}

issue-label-bot[bot] commented 3 years ago

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.84. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

marvinbuss commented 3 years ago

Hi @an185144, We are running unit and integration tests and I also have a separate repo, where I test all actions end to end and I was not able to reproduce the issue. Let me take a look tomorrow or next week and get back to you!

marvinbuss commented 3 years ago

Hi @an185144,

The issue you are describing sounds very similar to this one: https://github.com/Azure/aml-deploy/issues/41#issuecomment-731731871

Can you confirm that it is the same issue?

an185144 commented 3 years ago

Hi @marvinbuss - Thanks for looking into this. It does look very similar to the issue you mentioned above. I solved it by switching from 'v1' branch of aml-deploy to 'azureml_1_18 branch'. I think the issue is happening with the Dockerfile with the older version of the SDK.

It occurred to me to look at the Azure ML SDK version because a lot of the issues here are related to the SDK version and thankfully, you already had a branch with the change. I would be looking more into the conditions which result in the issue this week. Thanks for the help.

marvinbuss commented 3 years ago

Hi @an185144, Thanks for your reply and your investigation. I will release the new version later today. Please switch back to v1, once this is done.
Have a great day!

marvinbuss commented 3 years ago

Hi @an185144, I just released a new version. Please switch back to v1 tag, whenever you are ready. The latest release makes use of the latest azureml-sdk 1.18.0.