Azure / azure-cli

Azure Command-Line Interface
MIT License
3.96k stars 2.94k forks source link

Ability to redirect AZ CLI Extension installation through an artifactory or proxy #19587

Open ascott12humanacom opened 2 years ago

ascott12humanacom commented 2 years ago

Is your feature request related to a problem? Please describe. Internal access to repositories like pypi.org and pythonhosted.org are being blocked in many organizations, and access to them is being routed through artifactory platforms that proxy the repository.

Access for tools like pip can be altered using the index-url parameter. However, it appears that when an Azure CLI extension needs to fetch and install a third-part component, it doesn't honor the system pip config. We need a way to tell Az extension to fetch these components from a different spot. I have been unable to find documentation on how to do this, or if it's even possible right now, thus the feature request.

Describe the solution you'd like The ability to redirect AZ Cli to pull components via an artifactory or proxy such as JFrog. They typically create URLS in the form of https://\<user>:\<Password>@\<jfrog repo host>/artifactory/pypy.org-remote/simple/ Or some variation thereof.

Describe alternatives you've considered There are no alternatives. Either this capability is introduced, or AZ CLi Extension installations will not succeed inside organizations that use Artifactories.

Additional context Output of a current attempt to install the ssh extension. You'll see the 403 error when python tries to retrieve the cryptography module from pypi.org in it:

C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2>az extension add --name ssh --debug cli.knack.cli: Command arguments: ['extension', 'add', '--name', 'ssh', '--debug'] cli.knack.cli: init debug log: Enable color in terminal. Init colorama. cli.knack.cli: Event: Cli.PreExecute [] cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x037D5148>, <function OutputProducer.on_global_arguments at 0x03941FA0>, <function CLIQuery.on_global_arguments at 0x03967BB0>] cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate [] cli.azure.cli.core: Modules found from index for 'extension': ['azure.cli.command_modules.extension'] cli.azure.cli.core: Loading command modules: cli.azure.cli.core: Name Load Time Groups Commands cli.azure.cli.core: extension 0.008 1 7 cli.azure.cli.core: Total (1) 0.008 1 7 cli.azure.cli.core: Loaded 1 groups, 7 commands. cli.azure.cli.core: Found a match in the command table. cli.azure.cli.core: Raw command : extension add cli.azure.cli.core: Command table: extension add cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x03D22028>] cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\ads5176.azure\commands\2021-09-16.12-10-37.extension_add.13576.log'. az_command_data_logger: command args: extension add --name {} --debug cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0x03D82190>, <function register_global_query_examples_argument..register_query_examples at 0x03D92808>] cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad [] cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0x03D92850>, <function register_cache_arguments..add_cache_arguments at 0x03D928E0>] cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded [] cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [] cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x03951028>, <function CLIQuery.handle_query_parameter at 0x03967BF8>, <function register_global_query_examples_argument..handle_example_parameter at 0x03D82220>, <function register_ids_argument..parse_ids_arguments at 0x03D92898>] urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443 D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'aka.ms'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings urllib3.connectionpool: https://aka.ms:443 "GET /azure-cli-extension-index-v1 HTTP/1.1" 301 0 urllib3.connectionpool: Starting new HTTPS connection (1): azcliextensionsync.blob.core.windows.net:443 D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'azcliextensionsync.blob.core.windows.net'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings urllib3.connectionpool: https://azcliextensionsync.blob.core.windows.net:443 "GET /index1/index.json HTTP/1.1" 200 1251912 cli.azure.cli.core.extension._resolve: Candidates ['ssh-0.1.0-py3-none-any.whl', 'ssh-0.1.4-py3-none-any.whl', 'ssh-0.1.5-py3-none-any.whl', 'ssh-0.1.6-py3-none-any.whl'] cli.azure.cli.core.extension._resolve: Candidates ['ssh-0.1.0-py3-none-any.whl', 'ssh-0.1.4-py3-none-any.whl', 'ssh-0.1.5-py3-none-any.whl', 'ssh-0.1.6-py3-none-any.whl'] cli.azure.cli.core.extension._resolve: Candidates ['ssh-0.1.0-py3-none-any.whl', 'ssh-0.1.4-py3-none-any.whl', 'ssh-0.1.5-py3-none-any.whl', 'ssh-0.1.6-py3-none-any.whl'] cli.azure.cli.core.extension._resolve: Candidates ['ssh-0.1.6-py3-none-any.whl', 'ssh-0.1.5-py3-none-any.whl', 'ssh-0.1.4-py3-none-any.whl', 'ssh-0.1.0-py3-none-any.whl'] cli.azure.cli.core.extension._resolve: Choosing the latest of the remaining candidates. cli.azure.cli.core.extension._resolve: Chosen {'downloadUrl': 'https://azcliprod.blob.core.windows.net/cli-extensions/ssh-0.1.6-py3-none-any.whl', 'filename': 'ssh-0.1.6-py3-none-any.whl', 'metadata': {'azext.isPreview': True, 'azext.minCliCoreVersion': '2.4.0', 'classifiers': ['Development Status :: 4 - Beta', 'Intended Audience :: Developers', 'Intended Audience :: System Administrators', 'Programming Language :: Python', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'License :: OSI Approved :: MIT License'], 'extensions': {'python.details': {'contacts': [{'email': 'azpycli@microsoft.com', 'name': 'Microsoft Corporation', 'role': 'author'}], 'document_names': {'description': 'DESCRIPTION.rst'}, 'project_urls': {'Home': 'https://github.com/Azure/azure-cli-extensions/tree/master/src/ssh'}}}, 'extras': [], 'generator': 'bdist_wheel (0.30.0)', 'license': 'MIT', 'metadata_version': '2.0', 'name': 'ssh', 'run_requires': [{'requires': ['cryptography']}], 'summary': 'SSH into Azure VMs using RBAC and AAD OpenSSH Certificates', 'version': '0.1.6'}, 'sha256Digest': 'a4de20dd94051c4b9143805cb9531075f56df5f56106c6aaed5e2fd1ffd62c3d'} cli.azure.cli.core.extension.operations: Extension source is url? True cli.azure.cli.core.extension.operations: Downloading https://azcliprod.blob.core.windows.net/cli-extensions/ssh-0.1.6-py3-none-any.whl to C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl urllib3.connectionpool: Starting new HTTPS connection (1): azcliprod.blob.core.windows.net:443 D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\urllib3/connectionpool.py:1013: InsecureRequestWarning: Unverified HTTPS request is being made to host 'azcliprod.blob.core.windows.net'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings urllib3.connectionpool: https://azcliprod.blob.core.windows.net:443 "GET /cli-extensions/ssh-0.1.6-py3-none-any.whl HTTP/1.1" 200 16816 cli.azure.cli.core.extension.operations: Downloaded to C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl cli.azure.cli.core.extension.operations: Validating the extension C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl cli.azure.cli.core.extension.operations: Checksum of C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl is OK cli.azure.cli.core.extension.operations: Validation successful on C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl cli.azure.cli.core.extension.operations: Executing pip with args: ['install', '--target', 'C:\Users\ads5176\.azure\cliextensions\ssh', 'C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl'] cli.azure.cli.core.extension.operations: Running: ['C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe', '-m', 'pip', 'install', '--target', 'C:\Users\ads5176\.azure\cliextensions\ssh', 'C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir'] cli.azure.cli.core.extension.operations: Using pip 21.0.1 from C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\pip (python 3.8) Non-user install due to --prefix or --target option Created temporary directory: C:\Users\ads5176\AppData\Local\Temp\pip-target-q52ol8ee Created temporary directory: C:\Users\ads5176\AppData\Local\Temp\pip-ephem-wheel-cache-9l2s0f8i Created temporary directory: C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o Initialized build tracking at C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o Created build tracker: C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o Entered build tracker: C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o Created temporary directory: C:\Users\ads5176\AppData\Local\Temp\pip-install-4xyeji6w Processing c:\users\ads5176\appdata\local\temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl Added ssh==0.1.6 from file:///C:/Users/ads5176/AppData/Local/Temp/tmpsai7br11/ssh-0.1.6-py3-none-any.whl to build tracker 'C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o' Removed ssh==0.1.6 from file:///C:/Users/ads5176/AppData/Local/Temp/tmpsai7br11/ssh-0.1.6-py3-none-any.whl from build tracker 'C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o' 1 location(s) to search for versions of cryptography:

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\pip_internal\cli\base_command.py", line 189, in _main status = self.run(options, args) File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\pip_internal\cli\req_command.py", line 178, in wrapper return func(self, options, args) File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\pip_internal\commands\install.py", line 316, in run requirement_set = resolver.resolve( File "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\lib\site-packages\pip_internal\resolution\resolvelib\resolver.py", line 127, in resolve six.raise_from(error, e) File "", line 3, in raise_from pip._internal.exceptions.DistributionNotFound: No matching distribution found for cryptography Removed build tracker: 'C:\Users\ads5176\AppData\Local\Temp\pip-req-tracker-dbv0ho9o'

cli.azure.cli.core.extension.operations: Command '['C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe', '-m', 'pip', 'install', '--target', 'C:\Users\ads5176\.azure\cliextensions\ssh', 'C:\Users\ads5176\AppData\Local\Temp\tmpsai7br11\ssh-0.1.6-py3-none-any.whl', '-vv', '--disable-pip-version-check', '--no-cache-dir']' returned non-zero exit status 1. cli.azure.cli.core.extension.operations: Pip failed so deleting anything we might have installed at C:\Users\ads5176.azure\cliextensions\ssh cli.azure.cli.core.extension.operations: Failed to delete 'C:\Users\ads5176.azure\cliextensions\ssh': [WinError 3] The system cannot find the path specified: 'C:\Users\ads5176\.azure\cliextensions\ssh'. Retrying ... cli.azure.cli.core.extension.operations: Failed to delete 'C:\Users\ads5176.azure\cliextensions\ssh': [WinError 3] The system cannot find the path specified: 'C:\Users\ads5176\.azure\cliextensions\ssh'. Retrying ... cli.azure.cli.core.extension.operations: Failed to delete 'C:\Users\ads5176.azure\cliextensions\ssh': [WinError 3] The system cannot find the path specified: 'C:\Users\ads5176\.azure\cliextensions\ssh'. Retrying ... cli.azure.cli.core.extension.operations: Failed to delete 'C:\Users\ads5176.azure\cliextensions\ssh': [WinError 3] The system cannot find the path specified: 'C:\Users\ads5176\.azure\cliextensions\ssh'. You may try to delete it manually. cli.azure.cli.core.util: azure.cli.core.util.handle_exception is called with an exception: cli.azure.cli.core.util: Traceback (most recent call last): File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 657, in execute File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 720, in _run_jobs_serially File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 691, in _run_job File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 328, in call File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/extension/custom.py", line 16, in add_extension_cmd File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/operations.py", line 339, in add_extension File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/extension/operations.py", line 173, in _add_whl_ext knack.util.CLIError: An error occurred. Pip failed with status code 1. Use --debug for more information.

cli.azure.cli.core.azclierror: An error occurred. Pip failed with status code 1. Use --debug for more information. az_command_data_logger: An error occurred. Pip failed with status code 1. Use --debug for more information. cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x03D22148>] az_command_data_logger: exit code: 1 cli.main: Command ran in 9.123 seconds (init: 0.450, invoke: 8.673) telemetry.save: Save telemetry record of length 3175 in cache telemetry.check: Returns Positive. telemetry.main: Begin creating telemetry upload process. telemetry.process: Creating upload process: "C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry__init__.pyc C:\Users\ads5176.azure" telemetry.process: Return from creating process telemetry.main: Finish creating telemetry upload process.

C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2>

yonzhan commented 2 years ago

@jiasli for awareness