az sql db export fails in ubuntu version of PS but not windows

trumsey2005 commented 8 months ago

Describe the bug

Steps to reproduce Below is a copy of my ADO pipeline to export sql dbs to a blob storage bacpac file on a regular basis... when it is run from an ubuntu builder (or from ubuntu using wsl + powershell) it will fail and say that the blob already exists. Since this is creating a blob based on a timestamped file name, this is not possible. When it is run from a windows ado builder or windows with powershell 7 it works as expected.

pool: vmImage: 'windows-latest'

steps:

task: AzureCLI@2 displayName: Create Sql Backups inputs: azureSubscription: "AzureDevelopment" scriptType: 'pscore' scriptLocation: 'inlineScript' inlineScript: | $currentDate = Get-Date $timestamp = $currentDate.ToUniversalTime().ToString("yyyyMMddHHmmss") $expiryDate = $currentDate.ToUniversalTime().AddHours(24).ToString("yyyy-MM-ddTHH:mmZ") $storageAccount = "empactk8sqlbackups4af" $containerExists = az storage container exists --account-name $storageAccount --name $(resourceGroup) --auth-mode login

if ($containerExists.exists -eq $True) { Write-Output "The container exists" } else { Write-Output "Creating Container" az storage container create --account-name $storageAccount --name $(resourceGroup) --auth-mode login }

$sasToken = az storage container generate-sas --account-name $storageAccount --name $(resourceGroup) --permissions lacrw --expiry $expiryDate --auth-mode login --as-user $databaseList = az sql db list --server $(resourceGroup) --resource-group $(resourceGroup) | ConvertFrom-Json

foreach($database in ($databaseList | Where-Object {$.name -ne "master"})){ $blobName = "${timestamp}/$($database.name)${timestamp}.bacpac" Write-Output $blobName $storageUri = "https://$storageAccount.blob.core.windows.net/$(resourceGroup)/${blobName}" Write-Output $storageUri az sql db export --name $database.name --resource-group $(resourceGroup) --server $(resourceGroup) --admin-user $(sql-username) --admin-password $(sql-password) --storage-key $sasToken --storage-key-type SharedAccessKey --storage-uri $storageUri --no-wait }

Related command

az sql db export

Errors

(BlobAlreadyExist) The ImportExport operation failed because the storage blob already exists. Code: BlobAlreadyExist

Issue script & Debug output

DEBUG: cli.knack.cli: init debug log: Cannot enable color. DEBUG: cli.knack.cli: Event: Cli.PreExecute [] DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fc746c71440>, <function OutputProducer.on_global_arguments at 0x7fc746c09e40>, <function CLIQuery.on_global_arguments at 0x7fc746c43920>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] DEBUG: cli.azure.cli.core: Modules found from index for 'sql': ['azure.cli.command_modules.sql', 'azure.cli.command_modules.sqlvm'] DEBUG: cli.azure.cli.core: Loading command modules: DEBUG: cli.azure.cli.core: Name Load Time Groups Commands DEBUG: cli.azure.cli.core: sql 0.136 56 215 DEBUG: cli.azure.cli.core: sqlvm 0.067 4 20 DEBUG: cli.azure.cli.core: Total (2) 0.203 60 235 DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next'] DEBUG: cli.azure.cli.core: Loading extensions: DEBUG: cli.azure.cli.core: Name Load Time Groups Commands Directory DEBUG: cli.azure.cli.core: Total (0) 0.000 0 0
DEBUG: cli.azure.cli.core: Loaded 59 groups, 235 commands. DEBUG: cli.azure.cli.core: Found a match in the command table. DEBUG: cli.azure.cli.core: Raw command : sql db export DEBUG: cli.azure.cli.core: Command table: sql db export DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7fc745b30fe0>] DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/vsts/work/_temp/.azclitask/commands/2024-02-12.17-29-45.sql_db_export.1850.log'. INFO: az_command_data_logger: command args: sql db export --name {} --resource-group {} --server {} --admin-user {} --admin-password {} --storage-key {} --storage-key-type {} --storage-uri {} --no-wait --debug DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x7fc745b474c0>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x7fc745b9d1c0>, <function register_cache_arguments..add_cache_arguments at 0x7fc745b9d300>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fc746c09ee0>, <function CLIQuery.handle_query_parameter at 0x7fc746c439c0>, <function register_ids_argument..parse_ids_arguments at 0x7fc745b9d260>] DEBUG: cli.azure.cli.core.commands.client_factory: Getting management service client client_type=SqlManagementClient DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/vsts/work/_temp/.azclitask/service_principal_entries.json', encrypt=False DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/vsts/work/_temp/.azclitask/msal_token_cache.json', encrypt=False DEBUG: cli.azure.cli.core.auth.binary_cache: load: /home/vsts/work/_temp/.azclitask/msal_http_cache.bin DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None) DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/5b28166c-1b07-4c94-a114-4b907d61f0ad/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'} DEBUG: msal.application: Broker enabled? False DEBUG: msal.application: Region to be used: None DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.core.windows.net//.default',), kwargs={} DEBUG: msal.application: Cache hit an AT DEBUG: msal.telemetry: Generate or reuse correlation_id: 486191cc-6c81-4570-abe1-cc7d0c473d75 DEBUG: cli.azure.cli.core.sdk.policies: Request URL: 'https://management.azure.com/subscriptions/d2b947c2-fcab-4ee5-8d17-04752d378c9d/resourceGroups/k8-internal-dev/providers/Microsoft.Sql/servers/k8-internal-dev/databases/EMPACT.Thumbnails/export?api-version=2023-02-01-preview' DEBUG: cli.azure.cli.core.sdk.policies: Request method: 'POST' DEBUG: cli.azure.cli.core.sdk.policies: Request headers: DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '573' DEBUG: cli.azure.cli.core.sdk.policies: 'Accept': 'application/json' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-client-request-id': '5013156b-c9cc-11ee-b3cb-55e0398569ae' DEBUG: cli.azure.cli.core.sdk.policies: 'CommandName': 'sql db export' DEBUG: cli.azure.cli.core.sdk.policies: 'ParameterSetName': '--name --resource-group --server --admin-user --admin-password --storage-key --storage-key-type --storage-uri --no-wait --debug' DEBUG: cli.azure.cli.core.sdk.policies: 'User-Agent': 'AZURECLI/2.56.0 (DEB) azsdk-python-mgmt-sql/4.0.0b13 Python/3.11.5 (Linux-6.2.0-1019-azure-x86_64-with-glibc2.35) VSTS_c993a862-4fb8-4bbf-8a7b-2a74611c4f13_build_701_0' DEBUG: cli.azure.cli.core.sdk.policies: 'Authorization': '*****' DEBUG: cli.azure.cli.core.sdk.policies: Request body: DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443 DEBUG: urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions/d2b947c2-fcab-4ee5-8d17-04752d378c9d/resourceGroups/k8-internal-dev/providers/Microsoft.Sql/servers/k8-internal-dev/databases/EMPACT.Thumbnails/export?api-version=2023-02-01-preview HTTP/1.1" 400 124 DEBUG: cli.azure.cli.core.sdk.policies: Response status: 400 DEBUG: cli.azure.cli.core.sdk.policies: Response headers: DEBUG: cli.azure.cli.core.sdk.policies: 'Cache-Control': 'no-cache' DEBUG: cli.azure.cli.core.sdk.policies: 'Pragma': 'no-cache' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Length': '124' DEBUG: cli.azure.cli.core.sdk.policies: 'Content-Type': 'application/json; charset=utf-8' DEBUG: cli.azure.cli.core.sdk.policies: 'Expires': '-1' DEBUG: cli.azure.cli.core.sdk.policies: 'x-ms-request-id': '64159c90-056b-44cb-9209-b94164f350d6'

Expected behavior

az command run in powershell should work the same across operating systems

Environment Summary

Working: azure-cli 2.56.0 *

core 2.56.0 * telemetry 1.1.0

Extensions: k8s-extension 1.6.0

Dependencies: msal 1.24.0b2 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Users\Tyler.azure\cliextensions'

Python (Windows) 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:21:31) [MSC v.1936 32 bit (Intel)]

Legal docs and information: aka.ms/AzureCliLegal

Not-Working: azure-cli 2.57.0

core 2.57.0 telemetry 1.1.0

Extensions: k8s-extension 1.6.0

Dependencies: msal 1.26.0 azure-mgmt-resource 23.1.0b2

Python location '/opt/az/bin/python3' Extensions directory '/home/tyler/.azure/cliextensions'

Python (Linux) 3.11.7 (main, Jan 31 2024, 05:29:49) [GCC 11.4.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Additional context

No response

yonzhan commented 8 months ago

Thank you for opening this issue, we will look into it.

microsoft-github-policy-service[bot] commented 8 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureSQLGitHub.

microsoft-github-policy-service[bot] commented 8 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Azure / azure-cli