Az Network Manager Connectivity Configuration Create/Update Not Working for Cross-tenant Hub

Describe the bug

Context When 'az network manager connect-config create', they have the option to select the property '--connectivity-topology "HubAndSpoke"'.

When hub and spoke topology is used, the customer must provide a resource id for the hub. This looks something like '--hub resource-id="/subscriptions/b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc/resourceGroups/JaredTestRG/providers/Microsoft.Network/virtualNetworks/testGroupMemberPublish" resource-type="Microsoft.Network/virtualNetworks"'

This resource id has a linked access check on it from ARM, and the resource id can be a resource from a secondary tenant.

Bug If the resource is in a secondary tenant, it is expected that the CLI extension will recognize this behind the scenes, and fetch a secondary token to authorize for this tenant. We use the 'x-ms-authorization-auxiliary' to accomplish this. The command is currently not doing this, so all create commands with a cross-tenant hub fail from azure cli.

Note, your team has already implemented this functionality for 'az network manager group static-member create', and it is working as expected.

Related command

az network manager connect-config create

Errors

(LinkedAuthorizationFailed) The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'. Code: LinkedAuthorizationFailed Message: The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'.

This linked access check failure is not expected, as the client I'm using has permission on both tenants.

Issue script & Debug output

C:\Users\jaredgorthy\OneDrive - Microsoft\Desktop>az network manager connect-config create --configuration-name "testCrossTenantFeature" --description "hellow world" --applies-to-groups group-connectivity="None" is-global=false network-group-id="/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/networkGroups/test" use-hub-gateway=false --connectivity-topology "HubAndSpoke" --delete-existing-peering false --hub resource-id="/subscriptions/b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc/resourceGroups/JaredTestRG/providers/Microsoft.Network/virtualNetworks/testGroupMemberPublish" resource-type="Microsoft.Network/virtualNetworks" --is-global true --network-manager-name "jaredgorthy" --resource-group "jaredgorthy-testResources" --debug cli.knack.cli: Command arguments: ['network', 'manager', 'connect-config', 'create', '--configuration-name', 'testCrossTenantFeature', '--description', 'hellow world', '--applies-to-groups', 'group-connectivity=None', 'is-global=false', 'network-group-id=/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/networkGroups/test', 'use-hub-gateway=false', '--connectivity-topology', 'HubAndSpoke', '--delete-existing-peering', 'false', '--hub', 'resource-id=/subscriptions/b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc/resourceGroups/JaredTestRG/providers/Microsoft.Network/virtualNetworks/testGroupMemberPublish', 'resource-type=Microsoft.Network/virtualNetworks', '--is-global', 'true', '--network-manager-name', 'jaredgorthy', '--resource-group', 'jaredgorthy-testResources', '--debug'] cli.knack.cli: init debug log: Enable color in terminal. Enable VT mode. cli.knack.cli: Event: Cli.PreExecute [] cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x018DF7F8>, <function OutputProducer.on_global_arguments at 0x01A088E8>, <function CLIQuery.on_global_arguments at 0x01A296B8>] cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] cli.azure.cli.core: Modules found from index for 'network': ['azure.cli.command_modules.network', 'azure.cli.command_modules.privatedns', 'azext_network_manager'] cli.azure.cli.core: Loading command modules: cli.azure.cli.core: Name Load Time Groups Commands cli.azure.cli.core: network 0.925 115 453 cli.azure.cli.core: privatedns 0.073 14 60 cli.azure.cli.core: Total (2) 0.998 129 513 cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next'] cli.azure.cli.core: Loading extensions: cli.azure.cli.core: Name Load Time Groups Commands Directory cli.azure.cli.core: virtual-network-manager 0.930 12 28 C:\Users\jaredgorthy.azure\cliextensions\virtual-network-manager cli.azure.cli.core: Total (1) 0.930 12 28 cli.azure.cli.core: Loaded 139 groups, 541 commands. cli.azure.cli.core: Found a match in the command table. cli.azure.cli.core: Raw command : network manager connect-config create cli.azure.cli.core: Command table: network manager connect-config create cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x03D0BF28>] cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\jaredgorthy.azure\commands\2024-01-03.12-20-54.network_manager_connect-config_create.37440.log'. az_command_data_logger: command args: network manager connect-config create --configuration-name {} --description {} --applies-to-groups {} {} {} {} --connectivity-topology {} --delete-existing-peering {} --hub {} {} --is-global {} --network-manager-name {} --resource-group {} --debug cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x03D7C3E8>] cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x03D7C398>, <function register_cache_arguments..add_cache_arguments at 0x03D7C488>] cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x01A08938>, <function CLIQuery.handle_query_parameter at 0x01A29708>, <function register_ids_argument..parse_ids_arguments at 0x03D7C438>] az_command_data_logger: extension name: virtual-network-manager az_command_data_logger: extension version: 1.0.0 cli.azure.cli.core.auth.persistence: build_persistence: location='C:\Users\jaredgorthy\.azure\msal_token_cache.bin', encrypt=True cli.azure.cli.core.auth.binary_cache: load: C:\Users\jaredgorthy.azure\msal_http_cache.bin urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'} msal.application: Broker enabled? False cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} cli.azure.cli.core.auth.msal_authentication: UserCredential.get_token: scopes=('https://management.core.windows.net//.default',), claims=None, kwargs={} msal.application: Cache hit an AT msal.telemetry: Generate or reuse correlation_id: 63819bd6-fd7a-4c72-85f6-5d89a588e221 cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature?api-version=2022-01-01' cli.azure.cli.core.sdk.policies: Request method: 'PUT' cli.azure.cli.core.sdk.policies: Request headers: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json' cli.azure.cli.core.sdk.policies: 'Accept': 'application/json' cli.azure.cli.core.sdk.policies: 'Content-Length': '656' cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '978604be-aa75-11ee-8474-a4ae1284d41e' cli.azure.cli.core.sdk.policies: 'CommandName': 'network manager connect-config create' cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--configuration-name --description --applies-to-groups --connectivity-topology --delete-existing-peering --hub --is-global --network-manager-name --resource-group --debug' cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.55.0 (MSI) (AAZ) azsdk-python-core/1.26.0 Python/3.11.5 (Windows-10-10.0.22621-SP0)' cli.azure.cli.core.sdk.policies: 'Authorization': '*****' cli.azure.cli.core.sdk.policies: Request body: cli.azure.cli.core.sdk.policies: {"properties": {"appliesToGroups": [{"groupConnectivity": "None", "isGlobal": "false", "networkGroupId": "/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/networkGroups/test", "useHubGateway": "false"}], "connectivityTopology": "HubAndSpoke", "deleteExistingPeering": "False", "description": "hellow world", "hubs": [{"resourceId": "/subscriptions/b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc/resourceGroups/JaredTestRG/providers/Microsoft.Network/virtualNetworks/testGroupMemberPublish", "resourceType": "Microsoft.Network/virtualNetworks"}], "isGlobal": "True"}} urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 urllib3.connectionpool: https://management.azure.com:443 "PUT /subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature?api-version=2022-01-01 HTTP/1.1" 403 509 cli.azure.cli.core.sdk.policies: Response status: 403 cli.azure.cli.core.sdk.policies: Response headers: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache' cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache' cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8' cli.azure.cli.core.sdk.policies: 'Expires': '-1' cli.azure.cli.core.sdk.policies: 'x-ms-failure-cause': 'gateway' cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '83636791-3520-4ba2-acb1-7cf5d357f4df' cli.azure.cli.core.sdk.policies: 'x-ms-correlation-request-id': '83636791-3520-4ba2-acb1-7cf5d357f4df' cli.azure.cli.core.sdk.policies: 'x-ms-routing-request-id': 'WESTUS:20240103T202055Z:83636791-3520-4ba2-acb1-7cf5d357f4df' cli.azure.cli.core.sdk.policies: 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains' cli.azure.cli.core.sdk.policies: 'X-Content-Type-Options': 'nosniff' cli.azure.cli.core.sdk.policies: 'Date': 'Wed, 03 Jan 2024 20:20:55 GMT' cli.azure.cli.core.sdk.policies: 'Connection': 'close' cli.azure.cli.core.sdk.policies: 'Content-Length': '509' cli.azure.cli.core.sdk.policies: Response content: cli.azure.cli.core.sdk.policies: {"error":{"code":"LinkedAuthorizationFailed","message":"The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'."}} cli.azure.cli.core.azclierror: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 663, in execute File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 726, in _run_jobs_serially File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 697, in _run_job File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 333, in call File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "C:\Users\jaredgorthy.azure\cliextensions\virtual-network-manager\azext_network_manager\custom.py", line 94, in network_manager_connect_config_create return _ConnectConfigCreate(cli_ctx=cmd.cli_ctx)(command_args=connectivity_configuration) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_command.py", line 155, in call File "C:\Users\jaredgorthy.azure\cliextensions\virtual-network-manager\azext_network_manager\aaz\latest\network\manager\connect_config_create.py", line 33, in _handler self._execute_operations() File "C:\Users\jaredgorthy.azure\cliextensions\virtual-network-manager\azext_network_manager\aaz\latest\network\manager\connect_config_create.py", line 140, in _execute_operations self.ConnectivityConfigurationsCreateOrUpdate(ctx=self.ctx)() File "C:\Users\jaredgorthy.azure\cliextensions\virtual-network-manager\azext_network_manager\aaz\latest\network\manager\connect_config_create.py", line 164, in call return self.on_error(session.http_response) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/aaz/_operation.py", line 332, in on_error azure.core.exceptions.HttpResponseError: (LinkedAuthorizationFailed) The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'. Code: LinkedAuthorizationFailed Message: The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'.

cli.azure.cli.core.azclierror: (LinkedAuthorizationFailed) The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'. Code: LinkedAuthorizationFailed Message: The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'. az_command_data_logger: (LinkedAuthorizationFailed) The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'. Code: LinkedAuthorizationFailed Message: The client has permission to perform action 'Microsoft.Network/virtualNetworks/write' on scope '/subscriptions/f0dc2b34-dfad-40e4-83e0-2309fed8d00b/resourceGroups/jaredgorthy-testResources/providers/Microsoft.Network/networkManagers/jaredgorthy/connectivityConfigurations/testCrossTenantFeature', however the current tenant '72f988bf-86f1-41af-91ab-2d7cd011db47' is not authorized to access linked subscription 'b7a6ea8b-8aac-46e2-9cd9-0d22ff8778bc'. cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x03D0F0C8>] az_command_data_logger: exit code: 1 cli.main: Command ran in 4.794 seconds (init: 0.987, invoke: 3.807) telemetry.main: Begin splitting cli events and extra events, total events: 1 telemetry.client: Accumulated 0 events. Flush the clients. telemetry.main: Finish splitting cli events and extra events, cli events: 1 telemetry.save: Save telemetry record of length 4635 in cache telemetry.main: Begin creating telemetry upload process. telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry__init__.pyc C:\Users\jaredgorthy.azure" telemetry.process: Return from creating process telemetry.main: Finish creating telemetry upload process.

Expected behavior

I expect the command to recognize the '--hub resource-id="..."' is a resource from another tenant, and automatically fetch the token for that tenant behind the scenes. The token should then be added via an auxiliary header to ensure the linked access check passes.

Note, your team has already implemented this for 'az network manager group static-member create', since this can also contain references to resources in another tenant.

Environment Summary

C:\Users\jaredgorthy\OneDrive - Microsoft\Desktop>az --version azure-cli 2.55.0

core 2.55.0 telemetry 1.1.0

Extensions: account 0.2.5 azure-devops 0.26.0 virtual-network-manager 1.0.0

Dependencies: msal 1.24.0b2 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Users\jaredgorthy.azure\cliextensions' Development extension sources: C:\CLI_test\azure-cli-extensions

Python (Windows) 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:21:31) [MSC v.1936 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

C:\Users\jaredgorthy\OneDrive - Microsoft\Desktop>az extension update --name virtual-network-manager Latest version of 'virtual-network-manager' is already installed.

Additional context

This is blocking an s500 customer, please address asap.

Azure / azure-cli-extensions