Cannot install kubectl and kubelogin with az aks install-cli

[madacsbotond]

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name az aks install-cli

Errors:

The command failed with an unexpected error. Here is the traceback:
<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
Traceback (most recent call last):
  File "/opt/az/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/opt/az/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/az/lib/python3.10/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/az/lib/python3.10/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/az/lib/python3.10/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/opt/az/lib/python3.10/http/client.py", line 975, in send
    self.connect()
  File "/opt/az/lib/python3.10/http/client.py", line 1454, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/opt/az/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/opt/az/lib/python3.10/ssl.py", line 1071, in _create
    self.do_handshake()
  File "/opt/az/lib/python3.10/ssl.py", line 1342, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/acs/custom.py", line 1469, in k8s_install_cli
    k8s_install_kubectl(cmd, client_version, install_location, base_src_url, arch=arch)
  File "/opt/az/lib/python3.10/site-packages/azure/cli/command_modules/acs/custom.py", line 1591, in k8s_install_kubectl
    version = urlopen(source_url + '/stable.txt', context=context).read()
  File "/opt/az/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/az/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/opt/az/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/az/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/opt/az/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/opt/az/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• az aks install-cli

Expected Behavior

Environment Summary

Certificates are added to /opt/az/lib/python3.10/site-packages/certifi/cacert.pem
Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.31, Ubuntu 20.04.6 LTS
Python 3.10.10
Installer: DEB

azure-cli 2.48.1

Extensions:
azure-devops 0.26.0
ml 2.15.1

Dependencies:
msal 1.20.0
azure-mgmt-resource 22.0.0

Additional Context

I am using a proxy and vpn as well. My az login works as intended. I can use az aks install-cli in Powershell (which fails, but with HTTP Error 403: rate limit exceeded).

[yonzhan]

Thank you for opening this issue, we will look into it.

[jeffvan98]

Experiencing the same problem with new VM running Windows Server 2022 Datacenter Azure Edition; run az aks install-cli from command shell (not powershell). AZ CLI 2.48.1 installed via msi.

[madacsbotond]

I found topics about the same issue from some years ago, but the exact solution was never shown there. I had an issue with az login, but adding the proxy certificate fixed that.

[navba-MSFT]

@madacsbotond Apologies for the late reply. Thanks for reaching out to us and reporting this issue. From the error message it is pretty clear that there seems to be a proxy within your network. Please follow the below action plans to isolate if this is indeed related to the proxy.

Plan 1: Could you please try to run the same command from a different machine which is on a completely different network? and check if that works.

Plan 2: Try appending standard Python root CAs to your pem file and check if that helps to run the CLI command.

REQUESTS_CA_BUNDLE=$HOME/certs/internal.pem

Plan 3: Try disabling SSL certificate verification temporarily by setting the REQUESTS_CA_BUNDLE environment variable to None and check if that helps to run the CLI command.

export REQUESTS_CA_BUNDLE=None

az aks install-cli

Plan 4: Does other AzCLI commands like az login --debugcommands like work fine ?

Plan 5: Run this command and check the result:

python.exe" -c "import urllib.request; print(urllib.request.getproxies())"

Plan 6: Please follow this article to make sure the issue is not caused by Azure CLI, but by Python and proxy. https://github.com/jiasli/azure-notes/blob/master/cli/proxy-cert-linux.md

And to work behind a proxy follow this, please follow the instruction at https://docs.microsoft.com/cli/azure/use-cli-effectively#work-behind-a-proxy to trust the root certificate and add the root CA of the proxy to your CA bundle.

[navba-MSFT]

@madacsbotond I wanted to do quick follow-up to check if you had a chance to look at my above comment. Please let us know if you had any updates on this. Awaiting your reply.

[navba-MSFT]

@madacsbotond The action is pending on you to try the above suggestion. We will now proceed with closure of this GitHub issue. If you need any further assistance on this issue in future, please feel free to reopen this thread. We would be happy to help.