search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.45k stars 3k forks source link

`pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'` raised when started by a separate process #12841

Open ztgzest opened 1 month ago

ztgzest commented 1 month ago

Description

pip._vendor.distlib.scripts fails to locate finder for 'pip._vendor.distlib' when pip is loaded by a separate process like ddtrace-run

Expected behavior

pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib' should not be raised when importing pip._vendor.distlib.scripts

pip version

24.1.2

Python version

3.10

OS

13.5.2 (22G91)

How to Reproduce

  1. setup a python module that imports pip._vendor.distlib.scripts or something from it. You can simply put the following in an empty python file to replicate the bug: import pip._vendor.distlib.scripts
  2. run the python file using ddtrace-run or anything that will run the python file in a separate process $ ddtrace-run python replicate_pip_bug.py

Example: replicate_pip_bug.py:

import pip._vendor.distlib.scripts

$ ddtrace-run python replicate_pip_bug.py

Running the following succeeds and should be the expected behavior: $ python replicate_pip_bug.py

Output

$ ddtrace-run python replicate_pip_bug.py
Traceback (most recent call last):
  File "/Users/home/replicate_pip_bug.py", line 1, in <module>
    import pip._vendor.distlib.scripts
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/opt/homebrew/Caskroom/miniconda/base/envs/my_env/lib/python3.10/site-packages/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/my_env/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 66, in <module>
    for r in finder(distlib_package).iterator("")
  File "/opt/homebrew/Caskroom/miniconda/base/envs/my_env/lib/python3.10/site-packages/pip/_vendor/distlib/resources.py", line 332, in finder
    raise DistlibException('Unable to locate finder for %r' % package)
pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'

Code of Conduct

notatallshaw commented 1 month ago

Can you give more context on the use case, importing internal or vendor modules from pip is not supported, hence the _ in the name when you try to import.

ztgzest commented 1 month ago

Can you give more context on the use case, importing internal or vendor modules from pip is not supported, hence the _ in the name when you try to import.

Our use case is running ddtrace-run with a py file that runs uvicorn; i've just narrowed down it down to that specific call for replication: $ ddtrace-run python /opt/srv/runner.py

Traceback (most recent call last):

  File "/opt/srv/runner.py", line 32, in <module>

    uvicorn.run('my_module:create_app', factory=True, host=os.environ.get('API_HOST'), port=os.environ.get('API_PORT'), proxy_headers=True, forwarded_allow_ips='*', access_log=False)

  File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 577, in run

    server.run()

  File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run

    return asyncio.run(self.serve(sockets=sockets))

  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run

    return loop.run_until_complete(main)

  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete

  File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve

    await self._serve(sockets)

  File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve

    config.load()

  File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 434, in load

    self.loaded_app = import_from_string(self.app)

  File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string

    module = importlib.import_module(module_str)

  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "/usr/local/lib/python3.10/site-packages/my_module/__init__.py", line 14, in <module>

    from .app import create_app

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/my_module/app.py", line 21, in <module>

    from model_deploy import DeployManager, config_dict

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/model_deploy/__init__.py", line 8, in <module>

    from model_deploy.utils import LOGGER_NAME, get_version

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/model_deploy/utils/__init__.py", line 2, in <module>

    from .utils import *

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/model_deploy/utils/utils.py", line 23, in <module>

    from pip._internal.operations import freeze

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/freeze.py", line 11, in <module>

    from pip._internal.req.constructors import (

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 9, in <module>

    from .req_install import InstallRequirement

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 40, in <module>

    from pip._internal.operations.install.wheel import install_wheel

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>

    from pip._vendor.distlib.scripts import ScriptMaker

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 66, in <module>

    for r in finder(distlib_package).iterator("")

  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distlib/resources.py", line 332, in finder

    raise DistlibException('Unable to locate finder for %r' % package)

pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'
notatallshaw commented 1 month ago

Our use case is running ddtrace-run with a py file that runs uvicorn

Sorry, but I'm oblivious to ddtrace-run and only have passing familiarity with uvicorn, so my question is, what does that have to do with pip?

Looking at your traceback it appears ddtrace-run is dynamically loading modules, there is no reason to expect that to work 100% of the time, especially from inside pip._vendor, as it is unsupported to import externally, and any file or functionality not explicitly used by pip will not be tested.

While I'm not saying this is definitely not a pip issue, I think it would make sense to reach out to ddtrace-run for support first as they will have a much better understanding of what ddtrace-run is trying to do by dynamically loading modules from pip.