[24.1] self signed certificates are no longer loaded

rasmus91 commented 3 months ago

Description

I am using a gitlab python repo, that proxies pypi.

After upgrading to pip 24.1 the certificate I have installed in /usr/local/share/ca-certificates is no longer loaded it seems, to validate that source.

After going back to pip 24.0 It works just fine once again.

Expected behavior

I would expect no problem installing packages through a private repo for which I have appropriate certificates installed systemwide.

pip version

24.1

Python version

3.10

OS

Ubuntu 22.04

How to Reproduce

Install certificate in /usr/local/share/ca-certificates
run sudo update-ca-certificates
setup a source repo to your private repository
install a package from it

Output

(this is output from ansible, but its still pip)

Looking in indexes: https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple
Obtaining file:///var/local/sbat-man-server-dev
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
:stderr:   error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Looking in indexes: https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple
      WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))': /api/v4/groups/25/-/packages/pypi/simple/poetry-core/
      Could not fetch URL https://gitlab%2Bdeploy-token-3:****@gitlab.ourinstance.com/api/v4/groups/25/-/packages/pypi/simple/poetry-core/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='gitlab.ourinstance.com', port=443): Max retries exceeded with url: /api/v4/groups/25/-/packages/pypi/simple/poetry-core/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))) - skipping
      ERROR: Could not find a version that satisfies the requirement poetry-core (from versions: none)
      ERROR: No matching distribution found for poetry-core
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

Code of Conduct

[X] I agree to follow the PSF Code of Conduct.

sweetlilmre commented 3 months ago

I can confirm similar behaviour on windows. For 24.1 pip fails with certificate errors for a self-signed cert and only works by pointing REQUESTS_CA_BUNDLE to a bundle containing the self-signed CA. 24.0 requires --use-feature truststore but works as expected. I tested all non-beta versions from 23 onwards.

ichard26 commented 3 months ago

Hey all, apologies for the disruption. Pip by default does not use the system certificate store. It is an optional feature enabled by the integration of truststore which did get bumped in 24.1. cc @sethmlarson

sethmlarson commented 3 months ago

@rasmus91 @sweetlilmre Just to confirm, you're running into this issue without using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps.

rasmus91 commented 3 months ago

Not using that option, no.

fre. 21. jun. 2024 16.06 skrev Seth Michael Larson @.***

:

@sweetlilmre https://github.com/sweetlilmre Just to confirm, you're running into this issue without using the optional "Truststore" feature or while you're using it (ie --with-feature=truststore)? Have to clarify because it's not clear from your posted reproduction steps.

— Reply to this email directly, view it on GitHub https://github.com/pypa/pip/issues/12779#issuecomment-2182823669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARQ7C5RHVTYOACO42J24XDZIQXNXAVCNFSM6AAAAABJVQC7RSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBSHAZDGNRWHE . You are receiving this because you authored the thread.Message ID: @.***>

sweetlilmre commented 3 months ago

@sethmlarson, while using it:

versions 23.0 to 24.0 using --with-feature=truststore work (these are the versions I tested).
version 24.1 using --with-feature=truststore does not work and I need to have REQUESTS_CA_BUNDLE defined and pointing to a bundle containing the required CA.

Sorry for the ambiguity. How it ever worked for @rasmus91 without the feature enabled surprises me.

rasmus91 commented 3 months ago

Yes, it did until 24.1. i dont understand why it wouldn't when the CA cert is installed system wide?

sethmlarson commented 3 months ago

So I'm actually suspecting that a defect in Requests is the source of the issues here, pip uses Requests 2.32.0 but there have been 3 bugfix releases since that release to Requests with some relating to certificates.

notatallshaw commented 3 months ago

I've always wanted to know how pip's vendoring process worked, so I've created two branches to test if requests is the issue:

Branch with latest version of requests: pip install git+https://github.com/notatallshaw/pip@requests-2.32.3 Branch with version of requests pip 24.0 had: pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

Can someone please try their workflow that works on 24.0 but breaks on 24.1 on both of these branches and report the results for both please (may need to use --no-cache-dir and --force to make sure there's a network connetion). It would also be apprecative if you could give exact steps of your workflow.

ichard26 commented 3 months ago

Looking at the requests changelog, I strongly suspect that https://github.com/psf/requests/pull/6655 is implicated. https://github.com/psf/requests/pull/6716 which landed in requests v2.32.3 restores the ability to specify custom SSLContexts in sub-classes of HTTPAdapter which pip does have. https://github.com/pypa/pip/blob/2753c77e2b3999683dac5201deb69fbd6c31f667/src/pip/_internal/network/session.py#L290

Please try @notatallshaw's branches to help us determine whether requests is the root problem and upgrading requests fixes the bug. Thanks!

sweetlilmre commented 3 months ago

Let me preface this with: I am no python / pip expert, in fact I probably don't know what I'm doing. That said I ran the following tests:

Create 2 directories, 2.32.3 and 2.31.0
Create a venv in both
Install the appropriate branch of pip in each
Test

Results:

2.32.3:

pip --version pip 24.0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable Install: python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.32.3

pip --version pip 24.2.dev0 from C:\source\piptest\2.32.3\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:

pip install pip-system-certs FAILS (SSL cert errors)
pip --use-feature truststore install pip-system-certs SUCCEEDS

2.31.0

pip --version pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable Install: python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

pip --version pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check: pip install pip-system-certs FAILS (traceback)

  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\source\piptest\2.31.0\.venv\Scripts\pip.exe\__main__.py", line 7, in <module>
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\main.py", line 78, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\__init__.py", line 114, in create_command
    module = importlib.import_module(module_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\commands\install.py", line 15, in <module>
    from pip._internal.cli.req_command import (
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\cli\req_command.py", line 18, in <module>
    from pip._internal.index.collector import LinkCollector
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_internal\index\collector.py", line 31, in <module>
    from pip._vendor import requests
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\__init__.py", line 45, in <module>
    from .exceptions import RequestsDependencyWarning
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\exceptions.py", line 9, in <module>
    from .compat import JSONDecodeError as CompatJSONDecodeError
  File "C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip\_vendor\requests\compat.py", line 10, in <module>
    import chardet
ModuleNotFoundError: No module named 'chardet'

notatallshaw commented 3 months ago

Thanks for running the tests, I'll fix the issue with the 2.31.0 branch later today and confirm.

Am I correct in understanding though that the 2.32.3 branch now succeeds where pip 24.1 fails?

sweetlilmre commented 3 months ago

As per my tests, yes. When you have the 2.31.0 branch sorted I will test

notatallshaw commented 3 months ago

I fixed the 2.31.0 branch if you want to reinstall it and try, though it's a little less important seeing that 2.32.3 works, it at least prooves definitively it was a requests regression and unrelated to truststore.

sweetlilmre commented 3 months ago

I reran the 2.31.0 branch for completeness sake.

2.31.0

pip --version
pip 24.0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Set REQUESTS_CA_BUNDLE environment variable
Install:
python.exe -m pip install git+https://github.com/notatallshaw/pip@requests-2.31.0

pip --version
pip 24.2.dev0 from C:\source\piptest\2.31.0\.venv\Lib\site-packages\pip (python 3.12)

Clear REQUESTS_CA_BUNDLE environment variable

Check:
pip install pip-system-certs FAILS (SSL)

pip --use-feature truststore install pip-system-certs SUCCEEDS

sweetlilmre commented 3 months ago

As a final comment on this issue, --use-feature truststore as a default and not a feature would be great. Doesn't fix the wider certifi / requests certificate handling debacle, but certainly helps for bootstrapping python, especially for new users.

pfmoore commented 3 months ago

Doesn't fix the wider certifi / requests certificate handling debacle

While I'm sure you didn't mean to offend, "debacle" is a bit strong here. We published a beta with the new requests release, and didn't get any feedback on this issue. The pip maintainers themselves don't have access to environments that need this feature, and so we weren't able to trigger the issue ourselves. If you have any suggestions (or better still a PR!) for how we could test this within our test suite then that would be appreciated - although it's arguable that this is something that should be covered in the requests test suite, so maybe proposing something at their end would also be useful.

We try to avoid regressions like this, but there's a lot of pip functionality that's only used by particular parts of our user base, and often those users don't (for perfectly good reasons) get involved in testing and proving new releases for us. Negative feedback tends to reduce our motivation for taking risks in those areas, which in turn slows the pace of developments and improvements for those users. It's difficult to find a good balance.

sweetlilmre commented 3 months ago

@pfmoore first off apologies, this was in no way intended as a slight toward the pip or request maintainers. In fact the handling of this issue has been top notch with quick responses and the option of neophytes like myself to offer some value from clear directions and things to test.

The debacle I was referring to is the general certificate handling case in the python ecosystem. truststore is a massive leap forward in terms of an improvement, similar to the intent behind pip-system-certs etc.

Unfortunately I live in a corporate environment with SSL inspection. This causes havoc all the time with new starts and even folks that have come back to python after a while, or had to reinstall for whatever reason. REQUESTS_CA_BUNDLE is always quoted as the solution but works in only some cases (seems to work for python and pip, but with certifi we inevitably end up having to patch the CA bundle, pip-system-certs used to work but has stopped recently).

truststore or something like that, baked in would be a huge win for python as a whole and debacle was the word that (perhaps a little unfortunately) I chose to express my frustration with the state of play.

Again apologies, and thank you to you, and all involved in this work.

notatallshaw commented 3 months ago

FWIW, back in my large corporation days I had to teach people to use px-proxy: https://github.com/genotrance/px?tab=readme-ov-file#installation

Which I found as the most consistent workaround to Python http ecosystem vs. enterprise security. Although I didn't get to try truststore, I'm now at a relatively small company where I don't face super esoteric security protocols

sweetlilmre commented 3 months ago

@notatallshaw thank you, I'll take a look at that.

pypa / pip