pulp / pulp-oci-images

Containerfiles and other assets for building Pulp 3 OCI images
https://docs.pulpproject.org/pulp_oci_images/
30 stars 55 forks source link

SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA signature digest algorithm too weak #607

Closed razorbladex401 closed 6 months ago

razorbladex401 commented 6 months ago

Version 3.45.1 (All in one)

Describe the bug I'm not sure if this is only tied to the container image or if this is a pulpcore issue. Since I'm using the all in one container I figure I'd start here. I recently updated a test instance of Pulp from 3.37.0 to 3.45.1. When synchronizing with RedHat repositories we are getting [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA signature digest algorithm too weak error. This only occurs against RedHat repositories. The remotes are configured with the entitlement certificate, certificate key and cacert.

Full error is: "error": { "traceback": " File \"/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py\", line 60, in _execute_task\n result = func(*args, *kwargs)\n File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 479, in synchronize\n remote_url = fetch_remote_url(remote, url)\n File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 282, in fetch_remote_url\n get_repomd_file(remote, normalized_remote_url)\n File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py\", line 238, in get_repomd_file\n return downloader.fetch()\n File \"/usr/local/lib/python3.9/site-packages/pulpcore/download/base.py\", line 185, in fetch\n result = asyncio.get_event_loop().run_until_complete(self.run(extra_data=extra_data))\n File \"/usr/lib64/python3.9/asyncio/base_events.py\", line 647, in run_until_complete\n return future.result()\n File \"/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py\", line 269, in run\n return await download_wrapper()\n File \"/usr/local/lib/python3.9/site-packages/backoff/_async.py\", line 151, in retry\n ret = await target(args, **kwargs)\n File \"/usr/local/lib/python3.9/site-packages/pulpcore/download/http.py\", line 254, in download_wrapper\n return await self._run(extra_data=extra_data)\n File \"/usr/local/lib/python3.9/site-packages/pulp_rpm/app/downloaders.py\", line 114, in _run\n async with self.session.get(\n File \"/usr/local/lib64/python3.9/site-packages/aiohttp/client.py\", line 1187, in aenter\n self._resp = await self._coro\n File \"/usr/local/lib64/python3.9/site-packages/aiohttp/client.py\", line 574, in _request\n conn = await self._connector.connect(\n File \"/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py\", line 544, in connect\n proto = await self._create_connection(req, traces, timeout)\n File \"/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py\", line 909, in _createconnection\n , proto = await self._create_proxy_connection(req, traces, timeout)\n File \"/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py\", line 1357, in _create_proxy_connection\n return await self._start_tls_connection(\n File \"/usr/local/lib64/python3.9/site-packages/aiohttp/connector.py\", line 1118, in _start_tls_connection\n raise ClientConnectorCertificateError(req.connection_key, exc) from exc\n", "description": "Cannot connect to host cdn.redhat.com:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: CA signature digest algorithm too weak (_ssl.c:1129)')]"

If I were to use curl to test the entitlement certificates, I don't get any errors. So, this appears to be something with python perhaps?

[root@containername ~]# curl -x http://myproxy:3128 --cacert rhelca.pem --cert rhelcert.pem --key rhelkey.pem https://cdn.redhat.com/content/dist/rhel9/9/x86_64/baseos/os/ <!DOCTYPE html>

repository index

repository index


📂 Packages/
📂 repodata/
        
[root@containername ~]# **To Reproduce** Steps to reproduce the behavior: Configure an RPM remote with entitlement certificates. Attempt to synchronize against any of their repos, example: https://cdn.redhat.com/content/dist/rhel9/9/x86_64/baseos/os **Expected behavior** A clear and concise description of what you expected to happen. Pulp should be able to synchronize without error. **Additional context** Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.
dralley commented 6 months ago

I was unable to reproduce this in a development environment although I haven't tried the all-in-one container yet.

Where is the cacert sourced from?

Are you able to run update-crypto-policies --show inside the container and paste the results?

razorbladex401 commented 6 months ago

@dralley this is the RHSM CA cert that is under /etc/rhsm/ca on RHEL systems.

[root@containername /]# update-crypto-policies --show DEFAULT

I've been doing some further testing today and I found that if I used the RHSM ca cert from a RHEL 9 server and using newer entitlement certificates it worked. Comparing the old RHSM ca cert with the new I can see that the old cert signature algorithm of sha1WithRSAEncryption but from a RHEL 9 server it has sha384WithRSAEncryption.

With that said I was able to get this to work, at least in my testing so far. I created a new system on the RHEL customer portal,

So, I'm guessing that at some point between the container with Pulp 3.37.0 to the newer one with 3.46.0, ca certificates with sha1 signatures are no longer supported?

dralley commented 6 months ago

The difference is almost certainly that we recently changed our base image from CentOS 8 Stream to CentOS 9 Stream. The crypto policy in EL9 rejects SHA1-based signatures/certificates by default.

I'm pretty sure update-crypto-policies --set LEGACY would allow everything to work fine with the old certificate, however it sounds like you've got everything sorted out now.

Thanks for the report and the background research! Would you be OK with me closing this?

Though we should potentially document this caveat somewhere.

razorbladex401 commented 6 months ago

Yes, now that I understand what's going on this can be closed.

I agree that this should be documented somewhere.