ImportError: cannot import name 'deploy_config_dict_to_obj' from 'azureml._model_management._util'

[gingergenius]

The DevOps pipeline started to fail on 14.05 after no changes on my side.

I have a project based on the example from this repository. The automatic retraining pipeline worked flawlessly for months until on 14.05 the stage with the ACI deployment via ml model deploy started to fail. Here is the stacktrace:

az_command_data_logger : command args: ml model deploy -n {} --model {} --model {} --model {} --ic {} --dc {} -g {} -w {} --overwrite --debug --enable-app-insights {}
metadata file logging enabled - writing logs to '/__w/_temp/.azclitask/commands'.
Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f63213c3b00>]
cli.azure.cli.core.util : The command failed with an unexpected error. Here is the traceback:
cannot import name 'deploy_config_dict_to_obj' from 'azureml._model_management._util' (/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/_model_management/_util.py)
Traceback (most recent call last):
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/knack/cli.py", line 215, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 546, in execute
    self.commands_loader.load_arguments(command)
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/__init__.py", line 330, in load_arguments
command_table[command].load_arguments()  # this loads the arguments via reflection
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 309, in load_arguments
    super(AzCliCommand, self).load_arguments()
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/knack/commands.py", line 105, in load_arguments
    cmd_args = self.arguments_loader()
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/__init__.py", line 534, in default_arguments_loader
    op = handler or self.get_op_handler(operation, operation_group=kwargs.get('operation_group'))
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azure/cli/core/__init__.py", line 574, in get_op_handler
    op = import_module(mod_to_import)
  File "/usr/local/envs/mlopspython_ci/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/__w/_temp/.azclitask/cliextensions/azure-cli-ml/azext_ml/model.py", line 7, in <module>
    from ._util import cli_context, collect_input_models, file_stream_to_object, \
  File "/__w/_temp/.azclitask/cliextensions/azure-cli-ml/azext_ml/_util.py", line 54, in <module>
    from azureml._model_management._util import deploy_config_dict_to_obj
ImportError: cannot import name 'deploy_config_dict_to_obj' from 'azureml._model_management._util' (/usr/local/envs/mlopspython_ci/lib/python3.7/site-packages/azureml/_model_management/_util.py)
cli.azure.cli.core.util : 
To open an issue, please run: 'az feedback'

I was using the mlops/python:latest Docker image. I saw under https://mcr.microsoft.com/v2/mlops/python/tags/list that there are new tags "build-20210513.1" and "build-20210513.2", so I assumed that is when the breaking changes were introduced to the Docker image. So I changed the version of the image to the latest one before that, "build-20210421.1". The error persisted.

I also tried to change the version of azureml-sdk to 1.27.* in the .yml files with dependencies for the scoring image because the latest commit (ae60e489f0c658ba313e6e0020c61b40ffe3bdc9) does that. This also did not change anything.

The command for the deployment stage is

    - task: AzureCLI@1
      displayName: 'Deploy'
      inputs:
        azureSubscription: '$(WORKSPACE_SVC_CONNECTION)'
        scriptLocation: inlineScript
        inlineScript: |
          set -e

          export SUBSCRIPTION_ID=$(az account show --query id -o tsv)

          cd ./ntcmlops/scoring

          az extension add -n azure-cli-ml

          az ml model deploy  -n prod-$(ACI_DEPLOYMENT_NAME) --model PROD_$(MODEL_NAME):$(MODEL_VERSION) --model PROD_$(PROJECT_PATH_TO_GUID_MAPPING_NAME):$(MODEL_VERSION) --model PROD_$(VECTORIZER_NAME):$(MODEL_VERSION)  --ic inference_config.yml --dc deployment_config_aci.yml -g $(RESOURCE_GROUP) -w $(WORKSPACE_NAME) --overwrite --debug --enable-app-insights True --scoring-timeout-ms 2000

[j-so]

There appears to be an issue with using the newest version of azure-cli-ml. We have reverted to using 1.27.*: https://github.com/microsoft/MLOpsPython/pull/361/files

Make sure to switch az extension add -n azure-cli-ml -> az extension add --source https://azurecliext.blob.core.windows.net/release/azure_cli_ml-1.27.0-py3-none-any.whl --yes

The error you're seeing is probably do a version mismatch between the CLI and the SDK. You'll have to directly reference the old 1.27.0 CLI version to resolve this. I would also recommend managing your own Docker image to prevent this kind of thing. The mlops/python:latest image is maintained just for this repo, and things may break if you aren't in sync.

[gingergenius]

That worked, thanks a lot. I would happily manage my own Docker image if I had the Dockerfile from mlops/python:latest to use an example. Could you share it please?

[j-so]

Here's the guide. The dockerfile is here.