Missing APIs in Storage Management Client on Linux Environments

[Elsie4ever]

Describe the bug

When using self.storage_mgmt_client.storage_accounts.get_properties from the storage management client in our extension, we encountered the following error:

ModuleNotFoundError: No module named 'azure.mgmt.storage.v2022_09_01'

Upon inspecting the Azure CLI package, we noticed that version 2022_09_01 is missing. Below is a screenshot of the error:

This issue has been reproduced on both WSL Ubuntu and Azure Cloud Shell, using the latest Azure CLI version 2.63.0.

Related command

.

Errors

The command failed with an unexpected error. Here is the traceback: No module named 'azure.mgmt.storage.v2022_09_01' Traceback (most recent call last): File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke cmd_result = self.invocation.execute(args) File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 664, in execute raise ex File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 731, in _run_jobs_serially results.append(self._run_job(expanded_arg, cmd_copy)) File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 701, in _run_job result = cmd_copy(params) File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/init.py", line 334, in call return self.handler(*args, kwargs) File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler return op(command_args) File "/home/elsie/.azure/cliextensions/azure-iot-ops/azext_edge/edge/commands_schema.py", line 30, in create_registry return SchemaRegistries(cmd).create( File "/home/elsie/.azure/cliextensions/azure-iot-ops/azext_edge/edge/providers/orchestration/resources/schema_registries.py", line 80, in create storage_properties: dict = self.storage_mgmt_client.storage_accounts.get_properties( File "/usr/lib64/az/lib/python3.9/site-packages/azure/mgmt/storage/_storage_management_client.py", line 1096, in storage_accounts from .v2022_09_01.operations import StorageAccountsOperations as OperationClass ModuleNotFoundError: No module named 'azure.mgmt.storage.v2022_09_01'

Issue script & Debug output

Expected behavior

Expected to have same API versions in windows

Environment Summary

azure-cli 2.63.0

core 2.63.0 telemetry 1.1.0

Extensions: ai-examples 0.2.5 azure-iot-ops 0.7.0a1 ml 2.28.0 ssh 2.0.5

Dependencies: msal 1.30.0 azure-mgmt-resource 23.1.1

Python location '/usr/bin/python3.9' Extensions directory '/home/elsie/.azure/cliextensions' Extensions system directory '/usr/lib/python3.9/site-packages/azure-cli-extensions'

Python (Linux) 3.9.19 (main, Jul 31 2024, 03:47:41) [GCC 11.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[github-actions[bot]]

Here are some similar issues that might help you. Please check if they can solve your problem.

• 19625

[Elsie4ever]

Here are some similar issues that might help you. Please check if they can solve your problem.

• No module named 'azure.mgmt.storage.v2021_04_01 #19625

this is not quite similar issue, ours is more like a packaging issue

[calvinhzy]

Hi @Elsie4ever, the api version 2022-09-01 is there for azure-mgmt-storage version 21.2.0. CLI does not package the python SDK, we only require that version 21.2.0. So it is possible the environment is not setup completely. You can run pip install azure-mgmt-storage==21.2.0 --force-reinstall What functionality are you trying to do with an older api version of the SDK? You can also try using the latest api version which is 2023-05-01. Thanks.

[digimaun]

@calvinhzy Can you please clarify what you mean when you say

CLI does not package the python SDK

?

This is an issue that is impacting the packaged installs of Azure CLI, where azure cli dependencies are part of the packaged Azure CLI install.

Here is a concrete example in cloudshell

For some reason, only 4 API versions are available on azure-mgmt-storage. In cloudshell the az version is 2.63.0 which as you mentioned takes a dependency on azure-mgmt-storage==21.2.0 -- like your screenshot shows, this SDK version includes more than those 4 APIs in cloudshell from the prior screenshot.

Here is an example with azure CLI 2.53.0 MSI for Windows.

Now here is the same example with azure CLI 2.63.0 MSI for Windows

Note there are inconsistent API versions available.

As Az CLI extension developers, we support a range of older versions of Azure CLI rather than only latest. Our min CLI version is 2.53.0, so we pin the api_version of the storage client we use based on the version available in the min CLI version we support. This way we can insulate ourselves from potential breaking changes always using the latest API client.

Agree this looks like an environment setup issue, but it doesn't make sense to ask users to run pip commands against packaged CLI installs.

[calvinhzy]

We do not include all the old api versions for the SDK, otherwise the package would get increasingly bigger. If the extension requires it, it should be specified in the extension vendered sdks list https://github.com/Azure/azure-cli-extensions/tree/main/src/storage-preview/azext_storage_preview/vendored_sdks to include an older version of the azure-storage-mgmt sdk. But then it would be breaking as I see some commands already uses 2023-05-01. So what is breaking in version 2023-05-1 from 2022-09-01? Maybe using the new version works fine for your use case? Please give it a try.

[digimaun]

@calvinhzy, the fundamental issue is az cli extensions don't only support the latest azure cli. Az CLI release trains are different from CLI extension release trains. In this case our extension min CLI version is defined as 2.53.0 (and is a rolling window) which does not have 2023-05-01 (as you can see in the prior screenshot).

Furthermore, if Az CLI can remove SDK API versions on any given release (long term support or compatibility with extensions isn't defined), then extensions cannot depend on pinning to an API version of the shared dependency. Because of this, extensions will have additional overhead & complexity managing shared azure namespace dependencies independently. Sounds like embedding clients in these scenarios may be the workaround approach.

CC @yonzhan for awareness.

[calvinhzy]

@digimaun it seems like since CLI and extension do not have the same required API versions, the best way for now is to use the vendored sdk to include the necessary SDK versions for extension only. Thanks.