search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.96k stars 2.94k forks source link

`az storage blob copy start` Should not require to query the source storage account key #28167

Open tete17 opened 7 months ago

tete17 commented 7 months ago

Describe the bug

When performing an az storage blob copy start command to copy from one storage account to another with --auth-mode login & without --source-account-key the cli errors out unless the user has the capability to read the source storage account key.

This is because the cli uses the UploadFromURL api and creates a SAS url for the source blob. To do so it requires the storage account which if not provided queries it from azure directly.

Related command

az storage blob copy start \
     --auth-mode login \
     --source-account-name "${SOURCE_ACCOUNT_NAME}" \
     --source-container "${SOURCE_CONTAINER_NAME}" \
     --source-blob "${file}" \
     --account-name "${DESTINATION_ACCOUNT_NAME}"  \
     --destination-container "${DESTINATION_CONATINER_NAME}" \
     --destination-blob "${file}"

Errors

Source storage account "${SOURCE_ACCOUNT_NAME}" not found

Issue script & Debug output

cli.knack.cli: Command arguments: ['storage', 'blob', 'copy', 'start', '--source-account-name', 'REDACTED', '--account-name', 'REDACTED', '--source-blob', 'silly', '--source-container', 'silly', '--destination-container', 'silly', '--destination-blob', 'silly', '--auth-mode', 'login', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fa60e151360>, <function OutputProducer.on_global_arguments at 0x7fa60e024040>, <function CLIQuery.on_global_arguments at 0x7fa60e0512d0>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: storage                   0.049        57       269
cli.azure.cli.core: Total (1)                 0.049        57       269
cli.azure.cli.core: Loaded 57 groups, 269 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : storage blob copy start
cli.azure.cli.core: Command table: storage blob copy start
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fa60c785fc0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-01-10.19-15-46.storage_blob_copy_start.26.log'.
az_command_data_logger: command args: storage blob copy start --source-account-name {} --account-name {} --source-blob {} --source-container {} --destination-container {} --destination-blob {} --auth-mode {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7fa60c79ab00>]
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 641, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 641, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'ListKeyExpand'

cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 641, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2022_09_01.models' has no attribute 'CorsRuleAllowedMethodsItem'

cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7fa60c550a60>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7fa60c550b80>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fa60e0240d0>, <function CLIQuery.handle_query_parameter at 0x7fa60e051360>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fa60c550af0>]
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/service_principal_entries.json', encrypt=False
cli.azure.cli.core.auth.persistence: build_persistence: location='/root/.azure/msal_token_cache.json', encrypt=False
cli.azure.cli.core.auth.binary_cache: load: /root/.azure/msal_http_cache.bin
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
msal.application: Region to be used: None
cli.azure.cli.core.commands.client_factory: Getting management service client client_type=StorageManagementClient
urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9cf1b59a-47c7-4f73-bbfb-ec5f98db700f/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
msal.application: Broker enabled? False
msal.application: Region to be used: None
cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={}
msal.application: Cache hit an AT
msal.telemetry: Generate or reuse correlation_id: d14f5a8b-dd7b-4d90-8a30-22febb9650f3
cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/49523a5b-6879-448c-b2ab-ab8d5f2e9821/providers/Microsoft.Storage/storageAccounts?api-version=2022-09-01'
cli.azure.cli.core.sdk.policies: Request method: 'GET'
cli.azure.cli.core.sdk.policies: Request headers:
cli.azure.cli.core.sdk.policies:     'Accept': 'application/json'
cli.azure.cli.core.sdk.policies:     'x-ms-client-request-id': 'a85addf6-afec-11ee-9f09-ee34ae0ea202'
cli.azure.cli.core.sdk.policies:     'CommandName': 'storage blob copy start'
cli.azure.cli.core.sdk.policies:     'ParameterSetName': '--source-account-name --account-name --source-blob --source-container --destination-container --destination-blob --auth-mode --debug'
cli.azure.cli.core.sdk.policies:     'User-Agent': 'AZURECLI/2.48.1 (DOCKER) azsdk-python-azure-mgmt-storage/21.0.0 Python/3.10.11 (Linux-5.15.0-1051-azure-x86_64-with)'
cli.azure.cli.core.sdk.policies:     'Authorization': '*****'
cli.azure.cli.core.sdk.policies: Request body:
cli.azure.cli.core.sdk.policies: This request has no body
urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/49523a5b-6879-448c-b2ab-ab8d5f2e9821/providers/Microsoft.Storage/storageAccounts?api-version=2022-09-01 HTTP/1.1" 200 1497
cli.azure.cli.core.sdk.policies: Response status: 200
cli.azure.cli.core.sdk.policies: Response headers:
cli.azure.cli.core.sdk.policies:     'Cache-Control': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Pragma': 'no-cache'
cli.azure.cli.core.sdk.policies:     'Content-Type': 'application/json; charset=utf-8'
cli.azure.cli.core.sdk.policies:     'Expires': '-1'
cli.azure.cli.core.sdk.policies:     'x-ms-request-id': '1fdd5961-093e-4577-925f-cdf868dbff36'
cli.azure.cli.core.sdk.policies:     'x-ms-correlation-request-id': '1fdd5961-093e-4577-925f-cdf868dbff36'
cli.azure.cli.core.sdk.policies:     'x-ms-routing-request-id': 'WESTEUROPE:20240110T191548Z:1fdd5961-093e-4577-925f-cdf868dbff36'
cli.azure.cli.core.sdk.policies:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
cli.azure.cli.core.sdk.policies:     'X-Content-Type-Options': 'nosniff'
cli.azure.cli.core.sdk.policies:     'Date': 'Wed, 10 Jan 2024 19:15:47 GMT'
cli.azure.cli.core.sdk.policies:     'Content-Length': '1497'
cli.azure.cli.core.sdk.policies: Response content:
cli.azure.cli.core.sdk.policies: {"value":[{"sku":{"name":"Standard_ZRS","tier":"Standard"},"kind":"StorageV2","id":"/subscriptions/49523a5b-6879-448c-b2ab-ab8d5f2e9821/resourceGroups/storage/providers/Microsoft.Storage/storageAccounts/REDACTED","name":"REDACTED","type":"Microsoft.Storage/storageAccounts","location":"westeurope","tags":{},"properties":{"keyCreationTime":{"key1":null,"key2":null},"allowCrossTenantReplication":true,"privateEndpointConnections":[],"minimumTlsVersion":"TLS1_2","allowBlobPublicAccess":true,"allowSharedKeyAccess":true,"isHnsEnabled":false,"networkAcls":{"resourceAccessRules":[],"bypass":"AzureServices","virtualNetworkRules":[],"ipRules":[],"defaultAction":"Allow"},"supportsHttpsTrafficOnly":true,"encryption":{"services":{"file":{"keyType":"Account","enabled":true,"lastEnabledTime":"2020-02-06T10:35:41.8408699Z"},"blob":{"keyType":"Account","enabled":true,"lastEnabledTime":"2020-02-06T10:35:41.8408699Z"}},"keySource":"Microsoft.Storage"},"accessTier":"Hot","provisioningState":"Succeeded","creationTime":"2020-02-06T10:35:41.7627044Z","primaryEndpoints":{"dfs":"https://REDACTED.dfs.core.windows.net/","web":"https://REDACTED.z6.web.core.windows.net/","blob":"https://REDACTED.blob.core.windows.net/","queue":"https://REDACTED.queue.core.windows.net/","table":"https://REDACTED.table.core.windows.net/","file":"https://REDACTED.file.core.windows.net/"},"primaryLocation":"westeurope","statusOfPrimary":"available"}}]}
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 509, in validate_source_url
    source_account_key = _query_account_key(cmd.cli_ctx, source_account_name)
  File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 40, in _query_account_key
    rg, scf = _query_account_rg(cli_ctx, account_name)
  File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 58, in _query_account_rg
    raise ValueError("Storage account '{}' not found.".format(account_name))
ValueError: Storage account 'REDACTED' not found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 651, in execute
    self._validation(expanded_arg)
  File "/usr/local/lib/python3.10/site-packages/knack/invocation.py", line 113, in _validation
    self._validate_arg_level(parsed_ns)
  File "/usr/local/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 859, in _validate_arg_level
    validator(**self._build_kwargs(validator, ns))
  File "/usr/local/lib/python3.10/site-packages/azure/cli/command_modules/storage/_validators.py", line 511, in validate_source_url
    raise RequiredArgumentMissingError('Source storage account {} not found.'.format(source_account_name))
azure.cli.core.azclierror.RequiredArgumentMissingError: Source storage account REDACTED not found.

cli.azure.cli.core.azclierror: Source storage account REDACTED not found.
az_command_data_logger: Source storage account REDACTED not found.
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fa60c786200>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 1.600 seconds (init: 0.228, invoke: 1.372)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 3623 in cache
telemetry.check: Negative: The /root/.azure/telemetry.txt was modified at 2024-01-10 19:14:06.337305, which in less than 600.000000 s

Expected behavior

My expectation is that during the validation in src/azure-cli/azure/cli/command_modules/storage/_validators.py:507 the codes doesn't ask for the storage account key to create a SAS, but instead detect if the user is able to create a User delegation SAS using the existing login credentials and use that instead.

Allowing the command to succeed

Environment Summary

azure-cli 2.48.1 *

core 2.48.1 telemetry 1.0.8

Dependencies: msal 1.20.0 azure-mgmt-resource 22.0.0

Python location '/usr/local/bin/python' Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.10.11 (main, Apr 5 2023, 23:58:40) [GCC 12.2.1 20220924]

Legal docs and information: aka.ms/AzureCliLegal

You have 3 update(s) available. Consider updating your CLI installation with 'az upgrade'

Additional context

This is the real reason for https://github.com/Azure/azure-cli/issues/9003

One expect that by granting a UserManagedIdentity (in my particular case) the Storage Blob Data Reader would suffice to copy data from that storage. The problem is that this command secretly requires the secret key (the equivalent of root mode) to be able to create SASs.

With this change the aforementioned role would suffice again since it includes the ability to create user delegation SASs. This is specially important from a security perspective since I only want to give read access to my Identity and the secret key would expose me to a risk of privilege escalation very easily.

I have been trying to edit the code myself but I am not very familiar with python and the way the official documentation makes use of the SDK is very different from what this code does.

azure-client-tools-bot-prd[bot] commented 7 months ago

Hi @tete17,

2.48.1 is not the latest Azure CLI(2.56.0).

If you haven't already attempted to do so, please upgrade to the latest Azure CLI version by following https://learn.microsoft.com/en-us/cli/azure/update-azure-cli.

yonzhan commented 7 months ago

Thank you for opening this issue, we will look into it.

tete17 commented 5 months ago

Hey @calvinhzy did you had a chance to look into this?