search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4.01k stars 2.98k forks source link

`az keyvault key show` cannot reach keyvault using private endpoint on a different virtual network #28637

Open mangohawk opened 7 months ago

mangohawk commented 7 months ago

Describe the bug

I'm encountering an issue with accessing Azure Key Vault using a private endpoint from a different virtual network, despite both networks being peered. The configuration involves two virtual networks: one with the deployment script resource and the other with the Key Vault.

The issue arises when executing the command az keyvault key show --vault-name kv-key-test --name testing --query key.kid --debug to access a key within the Key Vault using the Bicep deploymentScripts@2023-08-01 resource. When the Key Vault and the deployment script are in the same virtual network, the command executes successfully. This success indicates that the private endpoint and the associated network configurations are correctly set up for private networking access.

However, the command fails to execute and times out when the Key Vault is in a different virtual network from the deployment script, even though the two virtual networks are peered. This behavior is unexpected since peered networks are supposed to facilitate seamless connectivity between resources as if they were within the same network.

I've demonstrated the issue in my GitHub repository using Bicep files:

works.bicep: Demonstrates a successful setup, where both the Key Vault and the container are in the same virtual network. bug.bicep: Shows the failing setup, where the Key Vault is in a separate virtual network from the deployment script.

Repository link: https://github.com/mangohawk/privatekeyvault

Related command

az keyvault key show --vault-name kv-key-test --name testing --query key.kid --debug

Errors

Deployment Script error:

undefined (Code: DeploymentScriptExceededMaxAllowedTime)

Issue script & Debug output

executionresult (bug.bicep).json:

{"ScriptStart":"2024-03-26 10:06:38 ","LoginAttempts":"0","LoginComplete":"2024-03-26 10:06:49 ","DownloadStart":"","DownloadComplete":"","UserScriptStart":"2024-03-26 10:06:49 ","UserScriptComplete":"2024-03-26 10:15:40 ","Result":"Failed","Error":"DEBUG: cli.knack.cli: Command arguments: ['keyvault', 'key', 'show', '--vault-name', 'kv-key-test', '--name', 'testing', '--query', 'key.kid', '--debug']\nDEBUG: cli.knack.cli: __init__ debug log:\nCannot enable color.\nDEBUG: cli.knack.cli: Event: Cli.PreExecute []\nDEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fd7c2a240e0>, <function OutputProducer.on_global_arguments at 0x7fd7c293a340>, <function CLIQuery.on_global_arguments at 0x7fd7c2973e20>]\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []\nDEBUG: cli.azure.cli.core: Modules found from index for 'keyvault': ['azure.cli.command_modules.keyvault']\nDEBUG: cli.azure.cli.core: Loading command modules:\nDEBUG: cli.azure.cli.core: Name Load Time Groups Commands\nDEBUG: cli.azure.cli.core: keyvault 0.012 20 113\nDEBUG: cli.azure.cli.core: Total (1) 0.012 20 113\nDEBUG: cli.azure.cli.core: Loaded 20 groups, 113 commands.\nDEBUG: cli.azure.cli.core: Found a match in the command table.\nDEBUG: cli.azure.cli.core: Raw command : keyvault key show\nDEBUG: cli.azure.cli.core: Command table: keyvault key show\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fd7c0cbbb00>]\nDEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-03-26.10-06-50.keyvault_key_show.110.log'.\nINFO: az_command_data_logger: command args: keyvault key show --vault-name {} --name {} --query {} --debug\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7fd7c0d04c20>]\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7fd7c0d07ec0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7fd7c0d34040>]\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []\nDEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fd7c293a3e0>, <function CLIQuery.handle_query_parameter at 0x7fd7c2973ec0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fd7c0d07f60>]\nDEBUG: urllib3.connectionpool: Starting new HTTP connection (1): 169.254.169.254:80\nDEBUG: urllib3.connectionpool: http://169.254.169.254:80 \"GET /metadata/identity/oauth2/token?resource=https%3A%2F%2Fmanagement.core.windows.net%2F&api-version=2018-02-01 HTTP/1.1\" 200 1923\nDEBUG: msrestazure.azure_active_directory: MSI: Retrieving a token from http://169.254.169.254/metadata/identity/oauth2/token, with payload {'resource': 'https://management.core.windows.net/', 'api-version': '2018-02-01'}\nDEBUG: msrestazure.azure_active_directory: MSI: Token retrieved\nDEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): kv-key-test.vault.azure.net:443\nDEBUG: urllib3.connectionpool: Starting new HTTPS connection (2): kv-key-test.vault.azure.net:443\nDEBUG: urllib3.connectionpool: Starting new HTTPS connection (3): kv-key-test.vault.azure.net:443\nDEBUG: urllib3.connectionpool: Starting new HTTPS connection (4): kv-key-test.vault.azure.net:443\nDEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py\", line 135, in keyvault_command_handler\n show_exception_handler(ex)\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/arm.py\", line 429, in show_exception_handler\n raise ex\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py\", line 112, in keyvault_command_handler\n result = op(**command_args)\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py\", line 78, in wrapper_use_tracer\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/keyvault/keys/_client.py\", line 385, in get_key\n bundle = self._client.get_key(self.vault_url, name, key_version=version or \"\", **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/tracing/decorator.py\", line 78, in wrapper_use_tracer\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/keyvault/keys/_generated/operations/_key_vault_client_operations.py\", line 1307, in get_key\n pipeline_response: PipelineResponse = self._client._pipeline.run( # pylint: disable=protected-access\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 211, in run\n return first_node.send(pipeline_request) # type: ignore\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 71, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 71, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 71, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n [Previous line repeated 2 more times]\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/policies/_redirect.py\", line 158, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/policies/_retry.py\", line 468, in send\n raise err\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/policies/_retry.py\", line 446, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/policies/_authentication.py\", line 118, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 71, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 71, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 71, in send\n response = self.next.send(request)\n ^^^^^^^^^^^^^^^^^^^^^^^\n [Previous line repeated 1 more time]\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/_base.py\", line 103, in send\n self._sender.send(request.http_request, **request.context.options),\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/core/pipeline/transport/_requests_basic.py\", line 361, in send\n raise error\nazure.core.exceptions.ServiceRequestError: (<urllib3.connection.HTTPSConnection object at 0x7fd7bef29d90>, 'Connection to kv-key-test.vault.azure.net timed out. (connect timeout=300)')\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/knack/cli.py\", line 233, in invoke\n cmd_result = self.invocation.execute(args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py\", line 663, in execute\n raise ex\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py\", line 726, in _run_jobs_serially\n results.append(self._run_job(expanded_arg, cmd_copy))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py\", line 697, in _run_job\n result = cmd_copy(params)\n ^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py\", line 333, in __call__\n return self.handler(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py\", line 138, in keyvault_command_handler\n return keyvault_exception_handler(self.command_loader, ex)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/keyvault/_command_type.py\", line 51, in keyvault_exception_handler\n raise CLIError(ex)\nknack.util.CLIError: (<urllib3.connection.HTTPSConnection object at 0x7fd7bef29d90>, 'Connection to kv-key-test.vault.azure.net timed out. (connect timeout=300)')\n\nERROR: cli.azure.cli.core.azclierror: (<urllib3.connection.HTTPSConnection object at 0x7fd7bef29d90>, 'Connection to kv-key-test.vault.azure.net timed out. (connect timeout=300)')\nERROR: az_command_data_logger: (<urllib3.connection.HTTPSConnection object at 0x7fd7bef29d90>, 'Connection to kv-key-test.vault.azure.net timed out. (connect timeout=300)')\nDEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fd7c0cbbd80>]\nINFO: az_command_data_logger: exit code: 1\nINFO: cli.__main__: Command ran in 530.693 seconds (init: 0.532, invoke: 530.161)\nINFO: telemetry.main: Begin splitting cli events and extra events, total events: 1\nINFO: telemetry.client: Accumulated 0 events. Flush the clients.\nINFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1\nINFO: telemetry.save: Save telemetry record of length 3582 in cache\nINFO: telemetry.main: Begin creating telemetry upload process.\nINFO: telemetry.process: Creating upload process: \"/usr/local/bin/python /usr/local/lib/python3.11/site-packages/azure/cli/telemetry/__init__.py /root/.azure\"\nINFO: telemetry.process: Return from creating process\nINFO: telemetry.main: Finish creating telemetry upload process."}

Expected behavior

executionresult (works.bicep).json:

{"ScriptStart":"2024-03-26 11:28:18 ","LoginAttempts":"0","LoginComplete":"2024-03-26 11:28:28 ","DownloadStart":"","DownloadComplete":"","UserScriptStart":"2024-03-26 11:28:28 ","UserScriptComplete":"2024-03-26 11:28:31 ","Result":"Succeeded"}

Environment Summary

azure-cli 2.54.0 bin dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var core 2.54.0 bin dev etc home lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var telemetry 1.1.0 Dependencies: msal 1.24.0b2 azure-mgmt-resource 23.1.0b2 Python location '/usr/local/bin/python' Extensions directory '/root/.azure/cliextensions' Python (Linux) 3.11.6 (main, Oct 19 2023, 05:43:48) [GCC 12.2.1 20220924] Legal docs and information: aka.ms/AzureCliLegal

Additional context

No response

azure-client-tools-bot-prd[bot] commented 7 months ago

Hi @mangohawk,

2.54.0 is not the latest Azure CLI(2.58.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

yonzhan commented 7 months ago

Thank you for opening this issue, we will look into it.

mangohawk commented 7 months ago

Hi @mangohawk,

2.54.0 is not the latest Azure CLI(2.58.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

Latest allowed version of Azure CLI is 2.54.0 when using deploymentscripts, which is why I am using it. Any later versions results in InvalidTemplateDeployment.

evelyn-ys commented 7 months ago

This sounds like issue from VNET peering service side, try Troubleshoot virtual network peering issues

From CLI's perspective, maybe you can try Configure Azure Key Vault networking settings