az postgres flexible-server execute.. no longer works

[cookiecurse]

Describe the bug

az postgres flexible-server execute no longer working due to pip version. https://learn.microsoft.com/en-us/cli/azure/postgres/flexible-server?view=azure-cli-latest#az-postgres-flexible-server-execute

Azure CLI version 2.58.0

Azure CLI extension: rdbms-connect version 1.0.4 Last updated 2 years ago: https://github.com/Azure/azure-cli-extensions/tree/main/src/rdbms-connect

Last time this command worked successfully was on February 7th 2024 that I know of with azure cli version 2.28.0

Related command

Example:

az postgres flexible-server execute -n -u -p -d postgres -q --output tsv

Errors

ERROR: cli.azure.cli.core.azclierror: An error occurred. Pip failed with status code 2. Use --debug for more information. ERROR: az_command_data_logger: An error occurred. Pip failed with status code 2. Use --debug for more information.

Issue script & Debug output

DEBUG: cli.knack.cli: Command arguments: ['postgres', 'flexible-server', 'execute', '-n', 'azr-', '-u', '', '-p', '', '-d', '', '-q', 'alter user *** with login;', '--output', 'tsv', '--debug'] DEBUG: cli.knack.cli: init debug log: Cannot enable color. DEBUG: cli.knack.cli: Event: Cli.PreExecute [] DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x01176CF8>, <function OutputProducer.on_global_arguments at 0x01495DE8>, <function CLIQuery.on_global_arguments at 0x014BFBB8>] DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] DEBUG: cli.azure.cli.core: Modules found from index for 'postgres': ['azure.cli.command_modules.rdbms'] DEBUG: cli.azure.cli.core: Loading command modules: DEBUG: cli.azure.cli.core: Name Load Time Groups Commands DEBUG: cli.azure.cli.core: rdbms 0.486 49 202 DEBUG: cli.azure.cli.core: Total (1) 0.486 49 202 DEBUG: cli.azure.cli.core: Loaded 49 groups, 202 commands. DEBUG: cli.azure.cli.core: Could not find a match in the command or command group table for 'postgres flexible-server execute'. The index may be outdated. DEBUG: cli.azure.cli.core: Loading all modules and extensions

TLDR

WARNING: cli.azure.cli.core.extension.dynamic_install: The command requires the extension rdbms-connect. It will be installed first. WARNING: cli.azure.cli.core.extension.operations: Default enabled including preview versions for extension installation now. Disabled in May 2024. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only. DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443 DEBUG: urllib3.connectionpool: https://aka.ms:443 "GET /azure-cli-extension-index-v1 HTTP/1.1" 301 0 DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): azcliextensionsync.blob.core.windows.net:443 DEBUG: urllib3.connectionpool: https://azcliextensionsync.blob.core.windows.net:443 "GET /index1/index.json HTTP/1.1" 200 3868308 DEBUG: cli.azure.cli.core.extension._resolve: Candidates ['rdbms_connect-0.1.0-py2.py3-none-any.whl', 'rdbms_connect-0.1.1-py2.py3-none-any.whl', 'rdbms_connect-0.1.2-py2.py3-none-any.whl', 'rdbms_connect-0.1.3-py2.py3-none-any.whl', 'rdbms_connect-0.1.4-py2.py3-none-any.whl', 'rdbms_connect-1.0.0-py2.py3-none-any.whl', 'rdbms_connect-1.0.1-py2.py3-none-any.whl', 'rdbms_connect-1.0.2-py2.py3-none-any.whl', 'rdbms_connect-1.0.3-py2.py3-none-any.whl', 'rdbms_connect-1.0.4-py2.py3-none-any.whl'] DEBUG: cli.azure.cli.core.extension._resolve: Candidates ['rdbms_connect-0.1.0-py2.py3-none-any.whl', 'rdbms_connect-0.1.1-py2.py3-none-any.whl', 'rdbms_connect-0.1.2-py2.py3-none-any.whl', 'rdbms_connect-0.1.3-py2.py3-none-any.whl', 'rdbms_connect-0.1.4-py2.py3-none-any.whl', 'rdbms_connect-1.0.0-py2.py3-none-any.whl', 'rdbms_connect-1.0.1-py2.py3-none-any.whl', 'rdbms_connect-1.0.2-py2.py3-none-any.whl', 'rdbms_connect-1.0.3-py2.py3-none-any.whl', 'rdbms_connect-1.0.4-py2.py3-none-any.whl'] DEBUG: cli.azure.cli.core.extension._resolve: Candidates ['rdbms_connect-0.1.0-py2.py3-none-any.whl', 'rdbms_connect-0.1.1-py2.py3-none-any.whl', 'rdbms_connect-0.1.2-py2.py3-none-any.whl', 'rdbms_connect-0.1.3-py2.py3-none-any.whl', 'rdbms_connect-0.1.4-py2.py3-none-any.whl', 'rdbms_connect-1.0.0-py2.py3-none-any.whl', 'rdbms_connect-1.0.1-py2.py3-none-any.whl', 'rdbms_connect-1.0.2-py2.py3-none-any.whl', 'rdbms_connect-1.0.3-py2.py3-none-any.whl', 'rdbms_connect-1.0.4-py2.py3-none-any.whl'] DEBUG: cli.azure.cli.core.extension._resolve: Candidates ['rdbms_connect-0.1.0-py2.py3-none-any.whl', 'rdbms_connect-0.1.1-py2.py3-none-any.whl', 'rdbms_connect-0.1.2-py2.py3-none-any.whl', 'rdbms_connect-0.1.3-py2.py3-none-any.whl', 'rdbms_connect-0.1.4-py2.py3-none-any.whl', 'rdbms_connect-1.0.0-py2.py3-none-any.whl', 'rdbms_connect-1.0.1-py2.py3-none-any.whl', 'rdbms_connect-1.0.2-py2.py3-none-any.whl', 'rdbms_connect-1.0.3-py2.py3-none-any.whl', 'rdbms_connect-1.0.4-py2.py3-none-any.whl'] DEBUG: cli.azure.cli.core.extension._resolve: Chosen {'downloadUrl': 'https://azcliprod.blob.core.windows.net/cli-extensions/rdbms_connect-1.0.4-py2.py3-none-any.whl', 'filename': 'rdbms_connect-1.0.4-py2.py3-none-any.whl', 'metadata': {'azext.minCliCoreVersion': '2.19.0', 'classifiers': ['Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'License :: OSI Approved :: MIT License'], 'extensions': {'python.details': {'contacts': [{'email': 'azpycli@microsoft.com', 'name': 'Microsoft Corporation', 'role': 'author'}], 'document_names': {'description': 'DESCRIPTION.rst'}, 'project_urls': {'Home': 'https://github.com/Azure/azure-cli-extensions/tree/main/src/rdbms-connect'}}}, 'extras': [], 'generator': 'bdist_wheel (0.30.0)', 'license': 'MIT', 'metadata_version': '2.0', 'name': 'rdbms-connect', 'run_requires': [{'requires': ['mycli (==1.26.1)', 'pgcli (==3.4.1)', 'psycopg2 (==2.9.3)', 'setproctitle (~=1.2.2)']}], 'summary': 'Support for testing connection to Azure Database for MySQL & PostgreSQL servers.', 'version': '1.0.4'}, 'sha256Digest': '8dc15a70881c71f21ba000854960b1be851a057b403dc8de156b73883efb543d'} DEBUG: cli.azure.cli.core.extension.operations: Extension source is url? True DEBUG: cli.azure.cli.core.extension.operations: Downloading https://azcliprod.blob.core.windows.net/cli-extensions/rdbms_connect-1.0.4-py2.py3-none-any.whl to C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): azcliprod.blob.core.windows.net:443 DEBUG: urllib3.connectionpool: https://azcliprod.blob.core.windows.net:443 "GET /cli-extensions/rdbms_connect-1.0.4-py2.py3-none-any.whl HTTP/1.1" 200 12935 DEBUG: cli.azure.cli.core.extension.operations: Downloaded to C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl DEBUG: cli.azure.cli.core.extension.operations: Validating the extension C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl DEBUG: cli.azure.cli.core.extension.operations: Checksum of C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl is OK DEBUG: cli.azure.cli.core.extension.operations: Validation successful on C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl DEBUG: cli.azure.cli.core.extension.operations: Executing pip with args: ['install', '--target', 'F:\DA_Def3\_work\_temp\.azclitask\cliextensions\rdbms-connect', 'C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl'] DEBUG: cli.azure.cli.core.extension.operations: Running: ['C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe', '-m', 'pip', 'install', '--target', 'F:\DA_Def3\_work\_temp\.azclitask\cliextensions\rdbms-connect', 'C:\Users\TFSPBW~2\AppData\Local\Temp\tmpaky9q9xx\rdbms_connect-1.0.4-py2.py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir'] DEBUG: cli.azure.cli.core.extension.operations: Using pip 24.0 from C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\pip (python 3.11)

File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/parser.py", line 281, in parse_known_args File "argparse.py", line 1902, in parse_known_args File "argparse.py", line 2096, in _parse_known_args File "argparse.py", line 2073, in consume_positionals File "argparse.py", line 1978, in take_action File "argparse.py", line 1241, in call File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/parser.py", line 281, in parse_known_args File "argparse.py", line 1902, in parse_known_args File "argparse.py", line 2096, in _parse_known_args File "argparse.py", line 2073, in consume_positionals File "argparse.py", line 1978, in take_action File "argparse.py", line 1241, in call File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/parser.py", line 281, in parse_known_args File "argparse.py", line 1902, in parse_known_args File "argparse.py", line 2096, in _parse_known_args File "argparse.py", line 2073, in consume_positionals File "argparse.py", line 1962, in take_action File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/parser.py", line 272, in _get_values File "argparse.py", line 2506, in _get_values File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/parser.py", line 300, in _check_value File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/dynamic_install.py", line 119, in try_install_extension File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/dynamic_install.py", line 227, in _check_value_in_extensions File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/operations.py", line 349, in add_extension File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/operations.py", line 172, in _add_whl_ext knack.util.CLIError: An error occurred. Pip failed with status code 2. Use --debug for more information.

ERROR: cli.azure.cli.core.azclierror: An error occurred. Pip failed with status code 2. Use --debug for more information. ERROR: az_command_data_logger: An error occurred. Pip failed with status code 2. Use --debug for more information. DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x02316C08>] INFO: az_command_data_logger: exit code: 1 INFO: cli.main: Command ran in 11.972 seconds (init: 0.436, invoke: 11.536) INFO: telemetry.main: Begin splitting cli events and extra events, total events: 1 INFO: telemetry.client: Accumulated 0 events. Flush the clients. INFO: telemetry.main: Finish splitting cli events and extra events, cli events: 1 INFO: telemetry.save: Save telemetry record of length 3845 in cache INFO: telemetry.main: Begin creating telemetry upload process. INFO: telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry__init__.pyc F:\DA_Def1_work_temp.azclitask" INFO: telemetry.process: Return from creating process INFO: telemetry.main: Finish creating telemetry upload process.

Expected behavior

az postgres flex exectute command runs without errors.

Environment Summary

azure-cli 2.58.0

core 2.58.0 telemetry 1.1.0

Extensions: azure-devops 1.0.0

Dependencies: msal 1.26.0 azure-mgmt-resource 23.1.0b2

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.11.7 (tags/v3.11.7:fa7a6f2, Dec 4 2023, 19:24:49) [MSC v.1937 64 bit (AMD64)]

Additional context

I have tried downgrading az CLI version to last working (difficult with self-hosted agent without permissions to do that), upgrading Python version, enabling preview on extensions and doublechecking that they are up to date, changing Build Agents even to Microsoft-hosted.

[yonzhan]

Thank you for opening this issue, we will look into it.

[microsoft-github-policy-service[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @sunilagarwal, @sr-msft, @niklarin.

[ManjulaChintharla]

Same issue .

[p6lidia]

@sunilagarwal @sr-msft @niklarin Do you by chance have any updates when this might be fixed? We cannot use az postgres commands at all and are forced to look at alternatives at this point.

[MDendura]

I've been having similar issues. Seems related to https://github.com/Azure/azure-cli/issues/27896 and https://github.com/Azure/azure-cli/issues/24494

[IamSiDean]

Everything works fine in a pipeline using an ubuntu-latest agent, but I can't get az postgres flexible-server commands to work in windows-latest at all because of the rdbms-connect issue mentioned above. Are the az postgres flexible-server commands not available at all in Windows now?

[IamSiDean]

Thank you for opening this issue, we will look into it.

@yonzhan Can you tell us if anyone is looking into this at the moment please?

[ViktorDimitrievski]

Same issue here. I can not use

az login --service-principal -u $servicePrincipalId -p $servicePrincipalKey --tenant $tenantId

az postgres flexible-server execute --name "$(dbcServer)" `
    --admin-user "$(dbcLogin)" --admin-password "$(dbcPassword)" `
    --database-name "$(dbcDatabase)" --file-path "$sqlScriptPath" `
    --extension-name rdbms-connect `
    --query-exit-on-error `
    --query-exit-on-stderr `
    --azcli-arg "--no-cache-dir --force-reinstall psycopg2-binary==2.9.1"

or

az postgres flexible-server execute --name "$(dbcServer)" `
    --admin-user "$(dbcLogin)" --admin-password "$(dbcPassword)" `
    --database-name "$(dbcDatabase)" --file-path "$sqlScriptPath"

What can we find as an alternative for this issue?