[Azure Arc] az k8s-runtime load-balancer enable failing with No module named 'azure.mgmt.kubernetesconfiguration'

[daniel-kajita]

Describe the bug

I am trying to do a quick set-up of Azure Arc with a kind Kubernetes and a personal Azure Arc. I am able to connect fine with it but at the time to follow this guide it is failing with :

cli.azure.cli.core: Unable to load extension 'k8s-runtime: No module named 'azure.mgmt.kubernetesconfiguration''. Use --debug for more information

I tried to make sure I installed the Python dependencies like specified here.

pip3 install azure-mgmt-kubernetesconfiguration
Requirement already satisfied: azure-mgmt-kubernetesconfiguration in /usr/local/lib/python3.11/site-packages (3.1.0) Requirement already satisfied: isodate<1.0.0,>=0.6.1 in /usr/local/lib/python3.11/site-packages (from azure-mgmt-kubernetesconfiguration) (0.6.1) Requirement already satisfied: azure-common~=1.1 in /usr/local/lib/python3.11/site-packages (from azure-mgmt-kubernetesconfiguration) (1.1.28) Requirement already satisfied: azure-mgmt-core<2.0.0,>=1.3.2 in /usr/local/lib/python3.11/site-packages (from azure-mgmt-kubernetesconfiguration) (1.4.0) Requirement already satisfied: azure-core<2.0.0,>=1.26.2 in /usr/local/lib/python3.11/site-packages (from azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (1.30.2) Requirement already satisfied: six in /usr/local/lib/python3.11/site-packages (from isodate<1.0.0,>=0.6.1->azure-mgmt-kubernetesconfiguration) (1.16.0) Requirement already satisfied: requests>=2.21.0 in /usr/local/lib/python3.11/site-packages (from azure-core<2.0.0,>=1.26.2->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (2.32.3) Requirement already satisfied: typing-extensions>=4.6.0 in /usr/local/lib/python3.11/site-packages (from azure-core<2.0.0,>=1.26.2->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (4.12.2) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.26.2->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (3.3.2) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.26.2->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (3.7) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.26.2->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (2.2.1) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/site-packages (from requests>=2.21.0->azure-core<2.0.0,>=1.26.2->azure-mgmt-core<2.0.0,>=1.3.2->azure-mgmt-kubernetesconfiguration) (2024.6.2)

Related command

az k8s-runtime --debug load-balancer enable --resource-uri subscriptions/XXXXXXXX/resourceGroups/YYYYYY/providers/Microsoft.Kubernetes/connectedClusters/ZZZZZZZZZ

Errors

az k8s-runtime --debug load-balancer enable --resource-uri subscriptions/XXXXXXXX/resourceGroups/YYYYYY/providers/Microsoft.Kubernetes/connectedClusters/ZZZZZZZZZ Unable to load extension 'k8s-runtime: No module named 'azure.mgmt.kubernetesconfiguration''. Use --debug for more information. 'k8s-runtime' is misspelled or not recognized by the system.

Examples from AI knowledge base: https://aka.ms/cli_ref Read more about the command in reference docs

Issue script & Debug output

az k8s-runtime --debug load-balancer enable --resource-uri subscriptions/XXXXXXXX/resourceGroups/YYYYYY/providers/Microsoft.Kubernetes/connectedClusters/ZZZZZZZZZ cli.knack.cli: Command arguments: ['k8s-runtime', '--debug', 'load-balancer', 'enable', '--resource-uri', 'subscriptions/XXXXXXXX/resourceGroups/YYYYYY/providers/Microsoft.Kubernetes/connectedClusters/ZZZZZZZZZ'] cli.knack.cli: init debug log: Enable color in terminal. cli.knack.cli: Event: Cli.PreExecute [] cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x1102da200>, <function OutputProducer.on_global_arguments at 0x110921580>, <function CLIQuery.on_global_arguments at 0x11094f240>] cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] cli.azure.cli.core: No module found from index for '['k8s-runtime', '--debug', 'load-balancer', 'enable', '--resource-uri', 'subscriptions/XXXXXXXX/resourceGroups/YYYYYY/providers/Microsoft.Kubernetes/connectedClusters/ZZZZZZZZZ']' cli.azure.cli.core: Loading all modules and extensions cli.azure.cli.core: Discovered command modules: ['acr', 'acs', 'advisor', 'ams', 'apim', 'appconfig', 'appservice', 'aro', 'backup', 'batch', 'batchai', 'billing', 'botservice', 'cdn', 'cloud', 'cognitiveservices', 'compute_recommender', 'config', 'configure', 'consumption', 'container', 'containerapp', 'cosmosdb', 'databoxedge', 'dla', 'dls', 'dms', 'eventgrid', 'eventhubs', 'extension', 'feedback', 'find', 'hdinsight', 'identity', 'interactive', 'iot', 'keyvault', 'kusto', 'lab', 'managedservices', 'maps', 'marketplaceordering', 'monitor', 'mysql', 'netappfiles', 'network', 'policyinsights', 'privatedns', 'profile', 'rdbms', 'redis', 'relay', 'resource', 'role', 'search', 'security', 'servicebus', 'serviceconnector', 'servicefabric', 'signalr', 'sql', 'sqlvm', 'storage', 'synapse', 'util', 'vm'] cli.azure.cli.core: Loading command modules: cli.azure.cli.core: Name Load Time Groups Commands cli.azure.cli.core: acr 0.119 36 149 cli.azure.cli.core: acs 0.027 14 76 cli.azure.cli.core: advisor 0.002 3 6 cli.azure.cli.core: ams 0.006 22 100 cli.azure.cli.core: apim 0.006 14 69 cli.azure.cli.core: appconfig 0.003 9 47 cli.azure.cli.core: appservice 0.053 79 270 cli.azure.cli.core: aro 0.009 1 10 cli.azure.cli.core: backup 0.003 16 60 cli.azure.cli.core: batch 0.023 34 102 cli.azure.cli.core: batchai 0.003 10 30 cli.azure.cli.core: billing 0.013 20 53 cli.azure.cli.core: botservice 0.003 12 42 cli.azure.cli.core: cdn 0.110 8 49 cli.azure.cli.core: cloud 0.002 1 7 cli.azure.cli.core: cognitiveservices 0.002 10 33 cli.azure.cli.core: compute_recommender 0.003 1 1 cli.azure.cli.core: config 0.001 2 7 cli.azure.cli.core: configure 0.001 2 5 cli.azure.cli.core: consumption 0.015 8 9 cli.azure.cli.core: container 0.008 1 11 cli.azure.cli.core: containerapp 0.084 36 115 cli.azure.cli.core: cosmosdb 0.013 58 199 cli.azure.cli.core: databoxedge 0.008 5 28 cli.azure.cli.core: dla 0.003 23 62 cli.azure.cli.core: dls 0.003 7 41 cli.azure.cli.core: dms 0.002 3 22 cli.azure.cli.core: eventgrid 0.004 25 96 cli.azure.cli.core: eventhubs 0.011 13 19 cli.azure.cli.core: extension 0.001 1 7 cli.azure.cli.core: feedback 0.001 1 2 cli.azure.cli.core: find 0.001 1 1 cli.azure.cli.core: hdinsight 0.006 8 39 cli.azure.cli.core: identity 0.002 2 11 cli.azure.cli.core: interactive 0.000 1 1 cli.azure.cli.core: iot 0.115 19 82 cli.azure.cli.core: keyvault 0.008 20 113 cli.azure.cli.core: kusto 0.002 3 14 cli.azure.cli.core: lab 0.003 11 34 cli.azure.cli.core: managedservices 0.001 3 8 cli.azure.cli.core: maps 0.001 5 13 cli.azure.cli.core: marketplaceordering 0.002 1 2 cli.azure.cli.core: monitor 0.426 17 61 cli.azure.cli.core: mysql 0.086 15 51 cli.azure.cli.core: netappfiles 0.038 8 17 cli.azure.cli.core: network 0.172 103 337 cli.azure.cli.core: policyinsights 0.016 9 17 cli.azure.cli.core: privatedns 0.023 14 60 cli.azure.cli.core: profile 0.002 2 8 cli.azure.cli.core: rdbms 0.021 49 202 cli.azure.cli.core: redis 0.003 7 38 cli.azure.cli.core: relay 0.022 7 8 cli.azure.cli.core: resource 0.011 51 231 cli.azure.cli.core: role 0.002 17 61 cli.azure.cli.core: search 0.007 7 19 cli.azure.cli.core: security 0.009 48 98 cli.azure.cli.core: servicebus 0.008 12 14 cli.azure.cli.core: serviceconnector 0.066 20 307 cli.azure.cli.core: servicefabric 0.015 27 80 cli.azure.cli.core: signalr 0.002 9 34 cli.azure.cli.core: sql 0.014 56 215 cli.azure.cli.core: sqlvm 0.024 4 20 cli.azure.cli.core: storage 0.039 59 273 cli.azure.cli.core: synapse 0.011 54 246 cli.azure.cli.core: util 0.001 3 7 cli.azure.cli.core: vm 0.052 58 233 cli.azure.cli.core: Total (66) 1.756 1205 4682 cli.azure.cli.core: Loading extensions: cli.azure.cli.core: Name Load Time Groups Commands Directory cli.azure.cli.core: connectedk8s 0.024 1 10 /Users/daniel/.azure/cliextensions/connectedk8s cli.azure.cli.core: customlocation 0.010 1 7 /Users/daniel/.azure/cliextensions/customlocation cli.azure.cli.core: k8s-extension 0.009 2 16 /Users/daniel/.azure/cliextensions/k8s-extension cli.knack.cli: Event: MainLoader.OnFailedExtensionLoad [] cli.azure.cli.core: Unable to load extension 'k8s-runtime: No module named 'azure.mgmt.kubernetesconfiguration''. Use --debug for more information. cli.azure.cli.core: Traceback (most recent call last): File "/usr/local/Cellar/azure-cli/2.61.0/libexec/lib/python3.11/site-packages/azure/cli/core/init.py", line 351, in _update_command_table_from_extensions _load_extension_command_loader(self, args, ext_mod) File "/usr/local/Cellar/azure-cli/2.61.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 1120, in _load_extension_command_loader return _load_command_loader(loader, args, ext, '') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/Cellar/azure-cli/2.61.0/libexec/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 1106, in _load_command_loader command_table = command_loader.load_command_table(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/.azure/cliextensions/k8s-runtime/azext_k8s_runtime/init.py", line 22, in load_command_table from azext_k8s_runtime.commands import load_command_table File "/Users/daniel/.azure/cliextensions/k8s-runtime/azext_k8s_runtime/commands.py", line 13, in from .custom import (enable_storage_class_cmd, disable_storage_class_cmd, File "/Users/daniel/.azure/cliextensions/k8s-runtime/azext_k8s_runtime/custom.py", line 13, in from .custom_commands.storage_class import enable_storage_class, disable_storage_class File "/Users/daniel/.azure/cliextensions/k8s-runtime/azext_k8s_runtime/custom_commands/storage_class.py", line 18, in from azure.mgmt.kubernetesconfiguration import SourceControlConfigurationClient ModuleNotFoundError: No module named 'azure.mgmt.kubernetesconfiguration'

cli.azure.cli.core: Total (3) 0.042 4 33
cli.azure.cli.core: Loaded 1195 groups, 4715 commands. cli.azure.cli.core: Updated command index in 0.004 seconds. cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x111634a40>] cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/daniel/.azure/commands/2024-06-13.21-29-45.unknown_command.31099.log'. az_command_data_logger: command args: k8s-runtime --debug {} {} --resource-uri {} cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x11169d6c0>] cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x11169d760>, <function register_cache_arguments..add_cache_arguments at 0x11169d8a0>] cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x113035a80>] cli.azure.cli.core.command_recommender: "--resource-uri" is an invalid parameter for command "k8s-runtime". urllib3.connectionpool: Starting new HTTPS connection (1): app.aladdin.microsoft.com:443 urllib3.connectionpool: https://app.aladdin.microsoft.com:443 "GET /api/v1.0/suggestions?query=%7B%22command%22%3A+%22k8s-runtime%22%2C+%22parameters%22%3A+%22%22%7D&clientType=AzureCli&context=%7B%22versionNumber%22%3A+%222.61.0%22%2C+%22errorType%22%3A+%22UnknownSubcommand%22%2C+%22correlationId%22%3A+%224352a3f9-9a5e-40df-80c6-251aca3929f3%22%2C+%22subscriptionId%22%3A+%224c2596b7-866e-4990-ab6d-ce2fdacc783e%22%2C+%22eventId%22%3A+%226ee4ee7e-e580-4349-9e83-62114de7c80a%22%7D HTTP/1.1" 200 None cli.azure.cli.core.command_recommender: "--resource-uri" is an invalid parameter for command "k8s-runtime". cli.azure.cli.core.azclierror: NoneType: None

cli.azure.cli.core.azclierror: 'k8s-runtime' is misspelled or not recognized by the system. az_command_data_logger: 'k8s-runtime' is misspelled or not recognized by the system.

Examples from AI knowledge base: https://aka.ms/cli_ref Read more about the command in reference docs cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x111634cc0>] az_command_data_logger: exit code: 2 cli.main: Command ran in 2.413 seconds (init: 0.138, invoke: 2.275) telemetry.main: Begin splitting cli events and extra events, total events: 1 telemetry.client: Accumulated 0 events. Flush the clients. telemetry.main: Finish splitting cli events and extra events, cli events: 1 telemetry.save: Save telemetry record of length 3864 in cache telemetry.main: Begin creating telemetry upload process. telemetry.process: Creating upload process: "/usr/local/Cellar/azure-cli/2.61.0/libexec/bin/python /usr/local/Cellar/azure-cli/2.61.0/libexec/lib/python3.11/site-packages/azure/cli/telemetry/init.py /Users/daniel/.azure" telemetry.process: Return from creating process telemetry.main: Finish creating telemetry upload process.

Expected behavior

I would expect the Azure Arc LoadBalancer to be enabled

Environment Summary

az --version azure-cli 2.61.0

core 2.61.0 telemetry 1.1.0

Extensions: connectedk8s 1.7.3 customlocation 0.1.3 k8s-extension 1.6.1 k8s-runtime 1.0.1

Dependencies: msal 1.28.0 azure-mgmt-resource 23.1.1

Python location '/usr/local/Cellar/azure-cli/2.61.0/libexec/bin/python' Extensions directory '/Users/daniel/.azure/cliextensions'

Python (Darwin) 3.11.9 (main, Apr 2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.3.9.4)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @dyu1208, @FumingZhang, @andyliuliming.

[FumingZhang]

Issue related to extension k8s-runtime. Could you please help take a look? +@ddadaal, @upxinxin

[HE-Xinyu]

hi Daniel,

Have you tried reinstalling the extension? It seems that the azure.mgmt.kubernetesconfiguration was dropped during the installation. Manually running pip install won't work because Azure CLI places dependencies in a different path.

[daniel-kajita]

hi Daniel,

Have you tried reinstalling the extension? It seems that the azure.mgmt.kubernetesconfiguration was dropped during the installation. Manually running pip install won't work because Azure CLI places dependencies in a different path.

Hello @HE-Xinyu,

I tried to uninstall and install it back, still have the same problem ! Adding the debug logs for removing and adding it back in an attached file, just in case if can help in your investigation.

az_extension_add_debug.txt az_extension_remove_debug.txt

Error persisted with :

Unable to load extension 'k8s-runtime: No module named 'azure.mgmt.kubernetesconfiguration''. Use --debug for more information.
'k8s-runtime' is misspelled or not recognized by the system.

Examples from AI knowledge base:
https://aka.ms/cli_ref
Read more about the command in reference docs

[wangzelin007]

@daniel-kajita The quantum issue has appeared before, but was closed without being resolved. https://github.com/Azure/azure-cli/issues/27452

[bebound]

I believe this is caused by the extension itself. For dependencies starting with azure-xxxx, the extension should vendor the dependencies instead of adding them into dependency list. https://github.com/Azure/azure-cli-extensions/blob/60c9eb0eb7aab91bfa6a227fc1bf4d60e5033aaf/src/k8s-runtime/setup.py#L30-L31

[wangzelin007]

The same goes for quantum extension https://github.com/Azure/azure-cli-extensions/blob/main/src/quantum/setup.py#L37 Hi @daniel-kajita You can ignore these two extensions until they vendored azure-related packages to their code and remove dependencies in setup.py.

[wangzelin007]

@HE-Xinyu @warren-jones Please vendor(copy) azure-related packages to your extensions. Otherwise, these extensions may not be available and online documentation may not be automatically updated.