Pants: cannot import name 'HTTPSHandler' from 'urllib.request'

[frickerj]

Describe the bug A clear and concise description of the bug.

I am operating in a corporate environment, using github actions.

When trying to run pants, I get this error:

stderr:
Could not find a compatible interpreter.

Examined the following working interpreters:
1.) /usr/bin/python3.10 CPython==3.10.12

Skipped the following broken interpreters:
1.) /actions-runner/install/_work/_tool/Python/3.11.7/x64/bin/python3.11:
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/atomic_directory.py", line 24, in <module>
    from pex.third_party import attr
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/third_party/__init__.py", line 614, in <module>
    install(expose=["attrs"])
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/third_party/__init__.py", line 581, in install
    VendorImporter.install_vendored(prefix=import_prefix(), root=root, expose=expose)
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/third_party/__init__.py", line 242, in install_vendored
    root = cls._abs_root(root)
           ^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/third_party/__init__.py", line 185, in _abs_root
    from pex import vendor
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/vendor/__init__.py", line 13, in <module>
    from pex.tracer import TRACER
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/tracer.py", line 12, in <module>
    from pex.variables import ENV
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/variables.py", line 20, in <module>
    from pex.orderedset import OrderedSet
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/orderedset.py", line 15, in <module>
    from pex.compatibility import MutableSet
  File "/root/.cache/pants/named_caches/pex_root/isolated/ac60ac27c85fa1751e8e471a569159e88a033b8f/pex/compatibility.py", line 130, in <module>
    from urllib.request import HTTPSHandler as HTTPSHandler
ImportError: cannot import name 'HTTPSHandler' from 'urllib.request' (/actions-runner/install/_work/_tool/Python/3.11.7/x64/lib/python3.11/urllib/request.py)

(See https://github.com/pantsbuild/pex/issues/1027 for a list of known breaks and workarounds.)

No working interpreter compatible with the requested constraints was found:

  Version matches CPython==3.11.*

I do not expect this error to occur. I have installed python3-distlib per the mentioned issue here: https://github.com/pantsbuild/pex/issues/1027

I have printed the content of the file (request.py), and I see this relevant section:

class HTTPHandler(AbstractHTTPHandler):

    def http_open(self, req):
        return self.do_open(http.client.HTTPConnection, req)

    http_request = AbstractHTTPHandler.do_request_

if hasattr(http.client, 'HTTPSConnection'):

    class HTTPSHandler(AbstractHTTPHandler):

        def __init__(self, debuglevel=0, context=None, check_hostname=None):
            AbstractHTTPHandler.__init__(self, debuglevel)
            self._context = context
            self._check_hostname = check_hostname

        def https_open(self, req):
            return self.do_open(http.client.HTTPSConnection, req,
                context=self._context, check_hostname=self._check_hostname)

        https_request = AbstractHTTPHandler.do_request_

    __all__.append('HTTPSHandler')

I'm wondering if this if hasattr(http.client, 'HTTPSConnection'): may return false, and thus I'm not able to import the HTTPSHandler.

Pants version Which version of Pants are you using?

2.19.0rc3

OS Are you encountering the bug on MacOS, Linux, or both?

Linux, gha

Additional info Add any other information about the problem here, such as attachments or links to gists, if relevant.

Pants.toml

[GLOBAL]
pants_version = "2.19.0rc3"
print_stacktrace = true

backend_packages.add = [
  "pants.backend.docker",
  "pants.backend.python",
  "pants.backend.build_files.fix.deprecations",
  "pants.backend.build_files.fmt.black",
  "pants.backend.python.lint.autoflake",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.docformatter",
  "pants.backend.python.lint.isort",
  "pants.backend.experimental.python.lint.ruff"
]
pants_ignore=[
    "**/proto/**",
]

[anonymous-telemetry]
enabled = false

[source]
root_patterns = ["/"]

[python]
interpreter_constraints = ["==3.11.*"]
enable_resolves = true
default_resolve = "python-default"

[python-repos]
indexes = [
# intentionally hidden
]

[python-bootstrap]
search_path = [
    "<PYENV>",
    "<PATH>",
    "/usr/local/bin",
    "/opt/homebrew/bin/python3"
]

[pex-cli]
version="v2.1.148"
url_template="file:%(buildroot)s/.github/pex"

[python.resolves]
python-default = "3rdparty/python/user_reqs.lock"
ingestion = "3rdparty/python/ingestion.lock"
dragonfish_dbt = "3rdparty/python/dragonfish_dbt.lock"
backfill = "3rdparty/python/backfill.lock"
services-raf = "3rdparty/python/services-raf.lock"
black = "3rdparty/python/black.lock"
autoflake = "3rdparty/python/autoflake.lock"
docformatter = "3rdparty/python/docformatter.lock"
isort = "3rdparty/python/isort.lock"
ruff = "3rdparty/python/ruff.lock"

[test]
use_coverage = true
report = true

[coverage-py]
report = ["xml", "html"]

[generate-lockfiles]
diff = true

# in order to use these tools, we need to create lockfiles
# so that the ci knows to download the requirements
# from artifactory, instead of pypi
# create lockfiles with `pants generate-lockfiles`
[black]
install_from_resolve = "black"

[autoflake]
install_from_resolve = "autoflake"

[docformatter]
install_from_resolve = "docformatter"

[isort]
install_from_resolve = "isort"

[ruff]
install_from_resolve = "ruff"

Github Action

  lint:
    name: ✅ Lint
    runs-on: [self-hosted, cosmos-prod]
    steps:
      - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4

# I don't think this step is actually required
      - name: Install python3-distlib
        run: |
          sudo apt-get update && sudo apt-get install -y build-essential
          sudo apt-get update && sudo apt-get install -y libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev

      - name: setup-python
        uses: anzx/cosmos-github-actions/setup-python@main
        with:
            python-version: '3.11'

      - name: Install python3-distlib
        run: sudo apt-get update && sudo apt-get install -y libpython3.11-dev python3-distlib

      - name: Initialize Pants
        uses: pantsbuild/actions/init-pants@main

      - name: ci
        shell: bash
        run: |
          pants lint ::

[jsirois]

You pretty much answered your own question here!

I'm wondering if this if hasattr(http.client, 'HTTPSConnection'): may return false, and thus I'm not able to import the HTTPSHandler.

Exactly. Search for HTTPSHandler here: https://docs.python.org/3.11/library/urllib.request.html

You find:

If the Python installation has SSL support (i.e., if the ssl module can be imported), HTTPSHandler will also be added.

Which links to: https://docs.python.org/3.11/library/ssl.html#module-ssl

This module provides access to Transport Layer Security (often known as “Secure Sockets Layer”) encryption and peer authentication facilities for network sockets, both client-side and server-side. This module uses the OpenSSL library. It is available on all modern Unix systems, Windows, macOS, and probably additional platforms, as long as OpenSSL is installed on that platform.

That seems to suggest you do not have OpenSSL (at least of the right version) installed.

You're on your own debugging this, but I'd suggest cutting out all middlemen and just replacing the Pants action stuff with a simple run step like so:

name: Sanity check Python 3.11 install
run: |
    python3.11 -c "import url"
    python3.11 -c "from urllib.request import HTTPSHandler"

Since the action you use to install Python is opaque here (I couldn't find it publicly anyhow), I can only suggest the result of an obvious search on issues importing ssl for Debian Python 3.11: https://github.com/pyenv/pyenv/issues/2760

(IOW: you may not be using pyenv at all behind the scenes and you're definitely not using an rpm based distro, but the issue details some OpenSSL version restrictions CPython 3.11 has)

[jsirois]

I've re-labelled this to a question since it seems to be a question about how to install CPython 3.11 with ssl support.

[jsirois]

One more homework helper: https://peps.python.org/pep-0644/

[frickerj]

Thanks for your huge help on this @jsirois, really appreciate it!

I've added this step to my action, per your message

      - name: Sanity check Python 3.11 install
        run: |
          python3.11 -c 'from urllib.request import HTTPSHandler; print("hello")'

Surprisingly, this step passes, while the error highlighted in the original post continues to occur when pex is trying to find an interpreter.

Step logs

Run python3.11 -c 'from urllib.request import HTTPSHandler; print("hello")'
  python3.11 -c 'from urllib.request import HTTPSHandler; print("hello")'
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /actions-runner/install/_work/_tool/Python/3.11.7/x64
    PKG_CONFIG_PATH: /actions-runner/install/_work/_tool/Python/3.11.7/x64/lib/pkgconfig
    Python_ROOT_DIR: /actions-runner/install/_work/_tool/Python/3.11.7/x64
    Python2_ROOT_DIR: /actions-runner/install/_work/_tool/Python/3.11.7/x64
    Python3_ROOT_DIR: /actions-runner/install/_work/_tool/Python/3.11.7/x64
    LD_LIBRARY_PATH: /actions-runner/install/_work/_tool/Python/3.11.7/x64/lib
hello

This also works: python3.11 -c 'from urllib.request import HTTPSHandler; import ssl; print( ssl.__file__ ); print("hello")'

I'm not exactly sure why this would happen, I'll continue to dig but it seems like the issue might be with pex?

[jsirois]

I'm not exactly sure why this would happen, I'll continue to dig but it seems like the issue might be with pex?

That still seems highly unlikely.

So ... debugging 101, and I'm guilty of a sloppy rec here, using things like python3.11 is imprecise. Again, your action to install Python is opaque to me, but you suspiciously sudo apt-get update && sudo apt-get install -y libpython3.11-dev ... above which implies the debian-based distro you use carries it own Python 3.11 in addition to whatever is in /actions-runner/install/_work/_tool/Python/3.11.7/x64. Have you tried which -a python3.11 to see if there are multiple Python 3.11's on the machine? Your OP error shows Pex has a problem with /actions-runner/install/_work/_tool/Python/3.11.7/x64/bin/python3.11 - to confirm your debugging run step is testing the same interpreter you might use that absolute path.

[frickerj]

I'm able to reproduce this error removing all apt-get, except this

      - name: install libpython3.11-dev
        run: sudo apt-get update && sudo apt-get install -y libpython3.11-dev

Without this step, I get this error:

Skipped the following broken interpreters:
1.) /actions-runner/install/_work/_tool/Python/3.11.7/x64/bin/python3.11:
/actions-runner/install/_work/_tool/Python/3.11.7/x64/bin/python3.11: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

This error surprises me because the file does exist, and is in the LD_LIBRARY_PATH.

I've confirmed via which -a that there is only one python3.11 installation.

Our setup-python action is just a wrapper around the github setup-python that also sets the registries to our internal mirrors.

I'm still quite confused why everything seems to work normally elsewhere but not with pants. Will continue to dig, really appreciate your help with this!

[jsirois]

@#$! Pants: LD_LIBRARY_PATH ... Pants is hermetic by default; that means it masks almost all env vars from processes it runs. It looks like your /actions-runner/install/_work/_tool/Python/3.11.7/x64/bin/python3.11 probably relies on LD_LIBRARY_PATH=/actions-runner/install/_work/_tool/Python/3.11.7/x64/lib being available in the environment for ld.so to find and link ssl at Python executable boot time. You might check the contents of the /actions-runner/install/_work/_tool/Python/3.11.7/x64/lib tree to see if it has what looks to be an open ssl .so in there to confirm my hunch, but you might also just plow ahead and configure https://www.pantsbuild.org/docs/reference-subprocess-environment#env_vars in a CI-specific pants.toml like so:

[subprocess-environment]
env_vars.add = ["LD_LIBRARY_PATH"]

Or else in a CI-specific env var like so (see: https://www.pantsbuild.org/docs/options#environment-variables-3) : PANTS_SUBPROCESS_ENVIRONMENT=LD_LIBRARY_PATH

[jsirois]

@frickerj I owe you a Coke, but yes - LD_LIBRARY_PATH is, in fact, the issue. Please report back with success or failure after ensuring Pants leaks LD_LIBRARY_PATH and we can close this as an answered question.

[frickerj]

[subprocess-environment]
env_vars.add = ["LD_LIBRARY_PATH"]

It works! Have tried a few times now and it has worked every time. Thanks mate, I really appreciate your help!

[jsirois]

I <3 Pants