search

Azure / azure-cli

Azure Command-Line Interface
MIT License
4k stars 2.98k forks source link

Failed to run: az storage blob list #28534

Open swotiendang opened 7 months ago

swotiendang commented 7 months ago

Describe the bug

I want to purge all file in azure storage container in batch, I used Azure cli to first list out file to be purged, then run purge command, but it failed

Related command

$ContainerName = "invoice" $StorageConnectionString = $CutOffDateString = "2022-07-01T00:00:00Z"

az storage blob list --container-name $ContainerName --connection-string $StorageConnectionString --query "[?properties.lastModified < '$CutOffDateString'][name,properties.lastModified]" --output json | Out-File -FilePath "./purging-files.txt"

az storage blob delete-batch --source $ContainerName --connection-string $StorageConnectionString --if-unmodified-since $CutOffDateString --delete-snapshots include

Errors

I cannot provide this log this time, in short, Remote network exception

Issue script & Debug output

DEBUG: cli.knack.cli: init debug log: Cannot enable color. DEBUG: cli.knack.cli: Event: Cli.PreExecute [] DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x0000020D0790B880>, <function OutputProducer.on_global_arguments at 0x0000020D07A96020>, <function CLIQuery.on_global_arguments at 0x0000020D07AC3BA0>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] DEBUG: cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage'] DEBUG: cli.azure.cli.core: Loading command modules: DEBUG: cli.azure.cli.core: Name Load Time Groups Commands DEBUG: cli.azure.cli.core: storage 0.469 59 273 DEBUG: cli.azure.cli.core: Total (1) 0.469 59 273 DEBUG: cli.azure.cli.core: Loaded 59 groups, 273 commands. DEBUG: cli.azure.cli.core: Found a match in the command table. DEBUG: cli.azure.cli.core: Raw command : storage blob list DEBUG: cli.azure.cli.core: Command table: storage blob list DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x0000020D0A966D40>] DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\roel.ricarte2.azure\commands\2024-03-08.08-06-36.storage_blob_list.7980.log'. INFO: az_command_data_logger: command args: storage blob list --debug --container-name {} --account-key {} --account-name {} --query {} --output {} DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x0000020D0A973380>] DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 660, in _get_attr AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType' DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 660, in _get_attr AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ListKeyExpand' DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 660, in _get_attr AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'CorsRuleAllowedMethodsItem' DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x0000020D0A9C1260>, <function register_cache_arguments..add_cache_arguments at 0x0000020D0A9C13A0>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0000020D07A960C0>, <function CLIQuery.handle_query_parameter at 0x0000020D07AC3C40>, <function register_ids_argument..parse_ids_arguments at 0x0000020D0A9C1300>] INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage auth_mode value from environment variables or config file. INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage sas_token value from environment variables or config file. DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): prodseafilestorage.blob.core.windows.net:443 DEBUG: urllib3.connectionpool: https://prodseafilestorage.blob.core.windows.net:443 "GET /invoice?restype=container&comp=list&maxresults=5000 HTTP/1.1" 200 None WARNING: cli.azure.cli.command_modules.storage.operations.blob: Next Marker: [3:11 PM] Gupta1, Sachin WARNING: cli.azure.cli.command_modules.storage.operations.blob: 2!96!MDAwMDI3ITAwMDk0NjAzNThfQk4yMzA5MDM5OTg1LnBkZiEwMDAwMjghOTk5OS0xMi0zMVQyMzo1OTo1OS45OTk5OTk5WiE-

DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x0000020D0A99E3E0>, <function

_x509_from_base64_to_hex_transform at 0x0000020D0A99E480>]

DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult [<function CLIQuery.handle_query_parameter..filter_output at 0x0000020D0D1F31A0>]

DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []

DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x0000020D0A966FC0>]

INFO: az_command_data_logger: exit code: 0

INFO: cli.main: Command ran in 8.953 seconds (init: 0.665, invoke: 8.287)

INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1

INFO: telemetry.client: Accumulated 0 events. Flush the clients.

INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1

INFO: telemetry.save: Save telemetry record of length 3557 in cache

INFO: telemetry.main: Begin creating telemetry upload process.

INFO: telemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files\Microsoft

SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry__init__.pyc C:\Users\roel.ricarte2.azure"

INFO: telemetry.process: Return from creating process

INFO: telemetry.main: Finish creating telemetry upload process.

Expected behavior

az storage blob list should be working and output the file to be purged before actually run

Environment Summary

azure-cli 2.58.0

core 2.58.0 telemetry 1.1.0

Dependencies: msal 1.26.0 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Users\TienDang.azure\cliextensions'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Additional context

It's working in my local machine for development storage container, but failed to run in production storage container The same az version, same script, but only failed in production

yonzhan commented 7 months ago

Thank you for opening this issue, we will look into it.

calvinhzy commented 7 months ago

Can you give the error code that it is throwing? You can manually redact out any sensitive info. Is the Remote network exception happening for the az storage blob list command? Can you try az login --tenant for the remote tenant?

swotiendang commented 7 months ago

@calvinhzy Yes, issue in azure storage blob list command Also try az login --tenant; but still error; full log here:

az : DEBUG: cli.knack.cli: Command arguments: ['storage', 'blob', 'list', '--debug', '--container-name', 'invoice', 
'--account-key', '', 
'--account-name', '', '--query', "[?properties.lastModified < 
'2022-07-01T00:00:00Z'][name,properties.lastModified]", '--output', 'json']
At line:6 char:1
+ az storage blob list --debug --container-name $ContainerName --accoun ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (DEBUG: cli.knac...utput', 'json']:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 
0x000001A64860B880>, <function OutputProducer.on_global_arguments at 0x000001A648796020>, <function 
CLIQuery.on_global_arguments at 0x000001A6487C3BA0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'storage': ['azure.cli.command_modules.storage']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: storage                   0.435        59       273
DEBUG: cli.azure.cli.core: Total (1)                 0.435        59       273
DEBUG: cli.azure.cli.core: Loaded 59 groups, 273 commands.
DEBUG: cli.azure.cli.core: Found a match in the command table.
DEBUG: cli.azure.cli.core: Raw command  : storage blob list
DEBUG: cli.azure.cli.core: Command table: storage blob list
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 
0x000001A64B6A6D40>]
DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 
'C:\Users\roel.ricarte2\.azure\commands\2024-03-12.03-34-11.storage_blob_list.8016.log'.
INFO: az_command_data_logger: command args: storage blob list --debug --container-name {} --account-key {} --account-name 
{} --query {} --output {}
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function 
register_global_subscription_argument.<locals>.add_subscription_parameter at 0x000001A64B6B3380>]
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 
660, in _get_attr
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 
660, in _get_attr
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ListKeyExpand'
DEBUG: cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/profiles/_shared.py", line 
660, in _get_attr
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'CorsRuleAllowedMethodsItem'
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function 
register_ids_argument.<locals>.add_ids_arguments at 0x000001A64B701260>, <function 
register_cache_arguments.<locals>.add_cache_arguments at 0x000001A64B7013A0>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 
0x000001A6487960C0>, <function CLIQuery.handle_query_parameter at 0x000001A6487C3C40>, <function 
register_ids_argument.<locals>.parse_ids_arguments at 0x000001A64B701300>]
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage auth_mode value from environment variables or 
config file.
INFO: cli.azure.cli.command_modules.storage._validators: Try to get storage sas_token value from environment variables or 
config file.
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): prodseafilestorage.blob.core.windows.net:443
DEBUG: urllib3.connectionpool: https://prodseafilestorage.blob.core.windows.net:443 "GET 
/invoice?restype=container&comp=list&maxresults=5000 HTTP/1.1" 200 None
WARNING: cli.azure.cli.command_modules.storage.operations.blob: Next Marker:
WARNING: cli.azure.cli.command_modules.storage.operations.blob: 
2!96!MDAwMDI3ITAwMDk0NjAzNThfMDAwNjQxODg3NTU5LnBkZiEwMDAwMjghOTk5OS0xMi0zMVQyMzo1OTo1OS45OTk5OTk5WiE-
DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x000001A64B6DE3E0>, 
<function _x509_from_base64_to_hex_transform at 0x000001A64B6DE480>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult [<function 
CLIQuery.handle_query_parameter.<locals>.filter_output at 0x000001A64DF231A0>]
DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x000001A64B6A6FC0>]
INFO: az_command_data_logger: exit code: 0
INFO: cli.__main__: Command ran in 9.050 seconds (init: 0.499, invoke: 8.550)
INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1
INFO: telemetry.client: Accumulated 0 events. Flush the clients.
INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1
INFO: telemetry.save: Save telemetry record of length 3557 in cache
INFO: telemetry.main: Begin creating telemetry upload process.
INFO: telemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program 
Files\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry\__init__.pyc C:\Users\roel.ricarte2\.azure"
INFO: telemetry.process: Return from creating process
INFO: telemetry.main: Finish creating telemetry upload process
calvinhzy commented 7 months ago

Hi @swotiendang I am suspecting the --query "[?properties.lastModified < '2022-07-01T00:00:00Z'][name,properties.lastModified]"is not finding the right blobs. Does it list all the files without the --query? And does it output everything to console with --output json | Out-File -FilePath "./purging-files.txt"

swotiendang commented 7 months ago

@calvinhzy the script is working fine in my machine, but it failed when ran in Bastion host for production storage. The script are same in both machine, same AzureCLI version; So, it should be no issue with script!

Karishma-Tiwari-MSFT commented 6 months ago

evelyn-ys , calvinhzy Would either of you be able to help with this issue? Another customer is seeing the same here: https://learn.microsoft.com/en-us/answers/questions/1652321/error-while-executing-cli-script-on-windows-vm-for