certificate verify failed

maxadamo commented 2 days ago

Although I have set NOMAD_CACERT, and my certificate is valid:

$ curl https://nomad-ui.mydomain.org:4646/
<a href="/ui/">Temporary Redirect</a>.

I get this error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='nomad-ui.mydomain.org', port=4646): Max retries exceeded with url: /v1/allocations?prefix=3147c68d&namespace=default (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)')))

Is this happening because I am not using NOMAD_CLIENT_CERT and NOMAD_CLIENT_KEY?

I am not sure, however, if I set NOMAD_SKIP_VERIFY (export NOMAD_SKIP_VERIFY=true), I still get this other warning message:

/Users/massimiliano.adamo/.local/pipx/venvs/nomad-tools/lib/python3.12/site-packages/urllib3/connectionpool.py:1099: InsecureRequestWarning: Unverified HTTPS request is being made to host 'nomad-ui.geant.org'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings

Hence I created a PR.

Kamilcuk commented 2 days ago

Hi. Our Nomad cluster does not use certificates, so I eyeballed the code and did not test it. It's this guy:

            verify=(
                False
                if NOMAD_SKIP_VERIFY in os.environ
                else os.environ[NOMAD_CACERT]
                if NOMAD_CACERT in os.environ
                else True
            ),

To confirm, did you set NOMAD_CACERT to the path (not content) of the certificate file?

Although I have set NOMAD_CACERT, and my certificate is valid:

I do not think this proves it, curl does not respect any NOMAD_* variables, it uses system certificates. Note that requests from python is also not curl, it uses REQUESTS_CA_BUNDLE. Python has its own certificate rules, and I have been bitten before by python not wanting to use proper certificate chain. See for example https://stackoverflow.com/questions/42982143/python-requests-how-to-use-system-ca-certificates-debian-ubuntu .

Kindly try:

 curl --cacert "$NOMAD_CACERT" https://nomad-ui.mydomain.org:4646/

But if plain old curl works for you, that means system certificate chain is ok, which suggests to me that you shoudl consider trying fixing python so it properly uses your system certificate chain.

Kindly try something along:

 python -c 'import requests; print(requests.get("https://nomad-ui.mydomain.org:4646/", verify="the/path/to/NOMAD/CACERT"))`

Is this happening because I am not using NOMAD_CLIENT_CERT and NOMAD_CLIENT_KEY?

No, these are different things.

When a client makes a connection, then the client can encrypt its connection using a pre-existing client certificate. The server needs CACERT to prove that client certificate is OK.

This does not affect the server. A server can respond using a different signed server certificate. Then the client needs CACERT to prove server response is OK.

maxadamo commented 2 days ago

Although I have set NOMAD_CACERT, and my certificate is valid:

I do not think this proves it, curl does not respect any NOMAD_* variables, it uses system certificates. Note that requests from python is also not curl, it uses REQUESTS_CA_BUNDLE.

Kindly try:
 curl --cacert "$NOMAD_CACERT" https://nomad-ui.mydomain.org:4646/

you're right. It proves that the certificate is valid, but it does not prove that NOMAD_CACERT is the proper CA for that certificate. Anyway curl seems to work even with NOMAD_CACERT:

$  curl --cacert "$NOMAD_CACERT" https://nomad-ui.mydomain.org:4646/
<a href="/ui/">Temporary Redirect</a>.

Kamilcuk commented 2 days ago

Great. So what about python? Kindly try something along:

python -c 'import os, requests; print(requests.get("https://nomad-ui.mydomain.org:4646/v1/allocations", verify=os.environ["NOMAD_CACERT"]))`

maxadamo commented 2 days ago

Great. So what about python? Kindly try something along:

python -c 'import os, requests; print(requests.get("https://nomad-ui.mydomain.org:4646/v1/allocations", verify=os.environ["NOMAD_CACERT"]))`

This is weird. Although the same CA works with curl, it's not working with Python requests. I am using a wildcard certificate, issued by SectiGO.

python3 -c 'import os, requests; print(requests.get("https://nomad-ui.mydomain.org:4646/v1/allocations", verify=os.environ["NOMAD_CACERT"]))'
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 1042, in _create
    self.do_handshake()
  File "/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 1320, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='nomad-ui.mydomain.org', port=4646): Max retries exceeded with url: /v1/allocations (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/homebrew/lib/python3.12/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='nomad-ui.mydomain.org', port=4646): Max retries exceeded with url: /v1/allocations (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get issuer certificate (_ssl.c:1000)')))

maxadamo commented 2 days ago

anyhow, the PR #8 covers the case when the certificate is self-signed, and IMO still makes sense, but we don't know why Python requests is not working with a proper CA. I can share the CA with your, but I can ensure that it works from any browser (Firefox, Chrome, Safari), curl, and it also works with the nomad CLI.

It does not work with you tool and it does not work with wget. But with wget I can get it to work:

wget --ca-certificate="$NOMAD_CACERT" https://nomad-ui.mydomain.org:4646

Kamilcuk commented 2 days ago

I found a comment https://stackoverflow.com/questions/30405867/how-to-get-python-requests-to-trust-a-self-signed-ssl-certificate#comment103135380_30405947 that the certificate has to have all in the chains. If not, I am out of ideas. My guesses are pip install certifi or pip install pip-system-certs, but that shouldn't change anything.

When my newborn gives me some time, I would want to create tests with self-signed certificates for this project, but dunno when it will happen.

edit: yup, you may have only intermediate CA in your cacert https://stackoverflow.com/questions/64219172/curl-cacert-vs-python-requests-verify . Download the full chain of the certificates. If you feel like it, post the cert,or just post the output of openssl x509 -text -noout -in, but if you have only one ---- PEM CERTIFICATE-thingy in your pem file, then it is not enough.

Kamilcuk / nomad-tools

certificate verify failed #7