jsirois
commented
2 months ago The switch from a bespoke resolver to using vendored Pip under the hood solved this issue for the most part. The vendored Pip and any bootstrapped Pip's do use their own vendored requests and Pex is left using its URLFetcher wrapper over urllib for these activities:
- Bootstrap Pip from https://bootstrap.pypa.io/pip/ when creating or updating a venv with an interpreter that does not support
-m ensurepip. (rare) - Fetching PEP-691 file hash metadata when creating locks via
pex3 lock {create,sync,update}. - Fetching the
sciencebinary from https://github.com/a-scie/lift/releases when building a native PEX (--scie {eager,lazy}. (~one time operation - cached) - Parsing
-r <URL>requirements, which preserves a Pip feature. (rare)
Of these, only the PEP-691 case is hit regularly for lock users. The discovered metadata is cached with no TTL; but any lock update using PyPI, which supports PEP-691 is going to exercise URLFetcher urllib code. That code does support timeouts as well as the equivalent of requests verify={True,/path/to/cert-bundle} and has now been battle tested for several years to be at least decent enough to not net bug reports. Since Pex has gone the route of being dependency free, that leaves vendoring a version of requests still compatible with Python 2.7; i.e. requests==2.27.1. That nets a ~600KB size-increase to the ~3.5MB Pex wheel. Although not unreasonable, there's just no current impetus to take on this dependency with the improvements to urllib use that have transpired.
This would mean either making requests an explicit dependency, or vendoring it. There's a bunch of stuff that requests gives us that is just plain smart (verify=True, more sensible retry policies, etc.) Right now we abstract away the smarts which makes it hard to implement features like #26.