search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.97k stars 2.95k forks source link

Login not required for az storage blob download --auth-mode login, but required for az storage fs directory download --auth-mode login? #29736

Open aj9411 opened 3 weeks ago

aj9411 commented 3 weeks ago

Describe the bug

Hey guys,

I am running a script on a build box, the script runs the following commands:

  1. az login --service-principal -u *** --tenant *** --allow-no-subscriptions --federated-token (works)
  2. az account set --subscription *** (works)
  3. az storage blob download --account-name *** -c *** -n *** -f *** --auth-mode login (works)
  4. az storage fs directory download --account-name *** --auth-mode login --file-system *** --source-path *** --destination-path . \ --recursive --only-show-errors (error)

The third command, az storage blob download, runs fine, it does not require a re-login. The fourth command though, az storage fs directory download, asks for login. I see the message:

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code *** to authenticate.

Naturally there's no login with the code, and the script ends up failing with:

failed to perform copy command due to error: Login Credentials missing. OAuth token, SAS token, or shared key should be provided for Blob FS ERROR: Failed to perform copy operation.

Why is az storage fs directory download asking for a new login? Shouldn't it use the logged-in identity? I previously used a user-delegated token to run az storage fs directory download, I raised this issue https://github.com/Azure/azure-cli/issues/29322 to get support for --auth-mode login. Thanks for adding support on version 2.63.0, it works fine when I run it from my PC, no re-login required. It's just when a build box is running this command. I do see the INFO log from the build box:

INFO: cli.azure.cli.command_modules.storage._validators: Cannot generate sas token. self.account_key should not be None., so perhaps that might be the cause?

Related command

az storage fs directory download

Errors

failed to perform copy command due to error: Login Credentials missing. OAuth token, SAS token, or shared key should be provided for Blob FS

Issue script & Debug output

DEBUG: cli.knack.cli: Command arguments: ['storage', 'fs', 'directory', 'download', '--account-name', ***, '--auth-mode', 'login', '--file-system', ***, '--source-path', ***, '--destination-path', '.', '--recursive', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x776ee0dba3a0>, <function OutputProducer.on_global_arguments at 0x776ee0ccff70>, <function CLIQuery.on_global_arguments at 0x776ee0c67550>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: storage                   0.161        59       273
DEBUG: cli.azure.cli.core: Total (1)                 0.161        59       273
DEBUG: cli.azure.cli.core: Loaded 59 groups, 273 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command  : storage fs directory download
DEBUG: cli.azure.cli.core: Command table: storage fs directory download
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x776ee012e4c0>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/__w/_temp/.azclitask/commands/2024-08-19.23-15-44.storage_fs_directory_download.557.log'.
INFO: az_command_data_logger: command args: storage fs directory download --account-name {} --auth-mode {} --file-system {} --source-path {} --destination-path {} --recursive --debug
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x776ee00cd0d0>]
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/profiles/_shared.py", line 660, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_05_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'

DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/profiles/_shared.py", line 660, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_05_01.models' has no attribute 'ListKeyExpand'

DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/profiles/_shared.py", line 660, in _get_attr
    op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_05_01.models' has no attribute 'CorsRuleAllowedMethodsItem'

DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x776ee00720d0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x776ee0091f70>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x776ee0c46040>, <function CLIQuery.handle_query_parameter at 0x776ee0c675e0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x776ee0091ee0>]
WARNING: This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/__w/_temp/.azclitask/service_principal_entries.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/__w/_temp/.azclitask/msal_token_cache.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.binary_cache: load: /__w/_temp/.azclitask/msal_http_cache.bin
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/***
DEBUG: msal.authority: openid_config("https://login.microsoftonline.com/***/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/***/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/***/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/***/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? None
DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://storage.azure.com/.default',), kwargs={}
DEBUG: msal.application: Cache hit an AT
DEBUG: msal.telemetry: Generate or reuse correlation_id: 39d2d72e-27d2-4a1f-ac3d-fe922efcc282
INFO: cli.azure.cli.command_modules.storage._validators: Cannot generate sas token. self.account_key should not be None.
WARNING: cli.azure.cli.command_modules.storage.azcopy.util: Azcopy not found, installing at /__w/_temp/.azclitask/bin/azcopy
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: Initializing with Entra authority: https://login.microsoftonline.com/***
DEBUG: msal.authority: openid_config("https://login.microsoftonline.com/***/v2.0/.well-known/openid-configuration") = {'token_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/***/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/***/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/***/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/***/kerberos', 'tenant_region_scope': 'WW', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? None
DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://storage.azure.com/.default',), kwargs={}
DEBUG: msal.application: Cache hit an AT
DEBUG: msal.telemetry: Generate or reuse correlation_id: 1948ccca-84f2-4aa6-aa4e-be9a11890548
WARNING: cli.azure.cli.command_modules.storage.azcopy.util: Azcopy command: ['/__w/_temp/.azclitask/bin/azcopy', 'copy', 'https://***.dfs.core.windows.net/***, '.', '--recursive']
INFO: Scanning...
INFO: azcopy: A newer version 10.26.0 is available to download

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code *** to authenticate.

failed to perform copy command due to error: Login Credentials missing. OAuth token, SAS token, or shared key should be provided for Blob FS
DEBUG: cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/usr/lib64/az/lib/python3.9/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 664, in execute
    raise ex
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 731, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 723, in _run_job
    return cmd_copy.exception_handler(ex)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/storage/__init__.py", line 430, in new_handler
    first(ex)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/storage/__init__.py", line 429, in new_handler
    raise ex
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 701, in _run_job
    result = cmd_copy(params)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/__init__.py", line 334, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/storage/operations/azcopy.py", line 86, in storage_fs_directory_copy
    azcopy.copy(source, destination, flags=flags)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 111, in copy
    self.run_command(['copy', source, destination] + flags)
  File "/usr/lib64/az/lib/python3.9/site-packages/azure/cli/command_modules/storage/azcopy/util.py", line 107, in run_command
    raise CLIError('Failed to perform {} operation.'.format(args[1]))
knack.util.CLIError: Failed to perform copy operation.

ERROR: cli.azure.cli.core.azclierror: Failed to perform copy operation.
ERROR: az_command_data_logger: Failed to perform copy operation.
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x776ee012e700>, <function _create_token_credential.<locals>._cancel_timer_event_handler at 0x776edd5cfca0>]
INFO: az_command_data_logger: exit code: 1
INFO: cli.__main__: Command ran in 905.019 seconds (init: 0.116, invoke: 904.904)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3887 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "/usr/bin/python3.9 /usr/lib/az/lib/python3.9/site-packages/azure/cli/telemetry/__init__.py /__w/_temp/.azclitask"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process.

Expected behavior

az storage fs directory download does not require re-login when running it from build box using service principal

Environment Summary

azure-cli 2.63.0

core 2.63.0 telemetry 1.1.0

Extensions: azure-devops 1.0.1 xsignextension 0.45

Dependencies: msal 1.30.0 azure-mgmt-resource 23.1.1

Python location '/usr/bin/python3.9' Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.9.19 (main, Jul 31 2024, 03:47:41) [GCC 11.2.0]

Additional context

No response

yonzhan commented 3 weeks ago

Thank you for opening this issue, we will look into it.

calvinhzy commented 3 weeks ago

I am thinking AzCopy version is not being updated to the latest if it works locally but not in a data box. Can you try manually updating AzCopy to 10.26.0? Will look into updating to a minimum version of AzCopy in CLI. Thanks.

aj9411 commented 3 weeks ago

@calvinhzy I tried with wget -O azcopy_v10.tar.gz https://aka.ms/downloadazcopy-v10-linux && tar -xf azcopy_v10.tar.gz --strip-components=1, but that didn't work. Any other suggestions?

edit: I am dumb, mixed up OS's. I was able to install the azcopy tool before running az storage fs directory download, I am fine now! There's a workaround for the problem, so I guess it is your decision if you want to keep this open