Open ztgzest opened 1 month ago
Can you give more context on the use case, importing internal or vendor modules from pip is not supported, hence the _
in the name when you try to import.
Can you give more context on the use case, importing internal or vendor modules from pip is not supported, hence the
_
in the name when you try to import.
Our use case is running ddtrace-run
with a py file that runs uvicorn; i've just narrowed down it down to that specific call for replication:
$ ddtrace-run python /opt/srv/runner.py
Traceback (most recent call last):
File "/opt/srv/runner.py", line 32, in <module>
uvicorn.run('my_module:create_app', factory=True, host=os.environ.get('API_HOST'), port=os.environ.get('API_PORT'), proxy_headers=True, forwarded_allow_ips='*', access_log=False)
File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 577, in run
server.run()
File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run
return asyncio.run(self.serve(sockets=sockets))
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve
await self._serve(sockets)
File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve
config.load()
File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 434, in load
self.loaded_app = import_from_string(self.app)
File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string
module = importlib.import_module(module_str)
File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/usr/local/lib/python3.10/site-packages/my_module/__init__.py", line 14, in <module>
from .app import create_app
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/my_module/app.py", line 21, in <module>
from model_deploy import DeployManager, config_dict
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/model_deploy/__init__.py", line 8, in <module>
from model_deploy.utils import LOGGER_NAME, get_version
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/model_deploy/utils/__init__.py", line 2, in <module>
from .utils import *
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/model_deploy/utils/utils.py", line 23, in <module>
from pip._internal.operations import freeze
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/freeze.py", line 11, in <module>
from pip._internal.req.constructors import (
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 9, in <module>
from .req_install import InstallRequirement
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 40, in <module>
from pip._internal.operations.install.wheel import install_wheel
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>
from pip._vendor.distlib.scripts import ScriptMaker
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module
self.loader.exec_module(module)
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 66, in <module>
for r in finder(distlib_package).iterator("")
File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distlib/resources.py", line 332, in finder
raise DistlibException('Unable to locate finder for %r' % package)
pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'
Our use case is running
ddtrace-run
with a py file that runs uvicorn
Sorry, but I'm oblivious to ddtrace-run
and only have passing familiarity with uvicorn, so my question is, what does that have to do with pip?
Looking at your traceback it appears ddtrace-run
is dynamically loading modules, there is no reason to expect that to work 100% of the time, especially from inside pip._vendor
, as it is unsupported to import externally, and any file or functionality not explicitly used by pip will not be tested.
While I'm not saying this is definitely not a pip issue, I think it would make sense to reach out to ddtrace-run
for support first as they will have a much better understanding of what ddtrace-run
is trying to do by dynamically loading modules from pip.
Description
pip._vendor.distlib.scripts
fails to locate finder for 'pip._vendor.distlib' whenpip
is loaded by a separate process likeddtrace-run
Expected behavior
pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'
should not be raised when importingpip._vendor.distlib.scripts
pip version
24.1.2
Python version
3.10
OS
13.5.2 (22G91)
How to Reproduce
pip._vendor.distlib.scripts
or something from it. You can simply put the following in an empty python file to replicate the bug:import pip._vendor.distlib.scripts
ddtrace-run
or anything that will run the python file in a separate process$ ddtrace-run python replicate_pip_bug.py
Example:
replicate_pip_bug.py
:$ ddtrace-run python replicate_pip_bug.py
Running the following succeeds and should be the expected behavior:
$ python replicate_pip_bug.py
Output
Code of Conduct