Open ztgzest opened 1 month ago
Thanks for the report, @ztgzest. We'll look into it.
We are facing a similar kind of problem when trying to run pip install
while we have datadog agent (setup with https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh) enabled to monitor python processes. The error we get is the following
(.venv) sh-4.2$ pip3 install -r requirements.txt
Traceback (most recent call last):
File "/app/.venv/bin/pip3", line 8, in <module>
sys.exit(main())
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 78, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
module = importlib.import_module(module_path)
File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 15, in <module>
from pip._internal.cli.req_command import (
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 19, in <module>
from pip._internal.index.package_finder import PackageFinder
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 31, in <module>
from pip._internal.req import InstallRequirement
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 9, in <module>
from .req_install import InstallRequirement
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 40, in <module>
from pip._internal.operations.install.wheel import install_wheel
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "/app/.venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>
from pip._vendor.distlib.scripts import ScriptMaker
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
self.loader.exec_module(module)
File "/app/.venv/lib/python3.8/site-packages/pip/_vendor/distlib/scripts.py", line 66, in <module>
for r in finder(distlib_package).iterator("")
File "/app/.venv/lib/python3.8/site-packages/pip/_vendor/distlib/resources.py", line 332, in finder
raise DistlibException('Unable to locate finder for %r' % package)
pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'
This started happening only recently and causing our services to fail.
Any updates on this?
Summary of problem
ddtrace-run
fails to run aFastAPI
app served byuvicorn
It's specifically failing to load and execute modules needed by the app. When running the app without
ddtrace-run
(python /opt/srv/runner.py
), it succeedsWhich version of dd-trace-py are you using?
2.9.2
Which version of pip are you using?
24.2.1
Which libraries and their versions are you using?
`pip freeze`
annotated-types==0.6.0 anyio==3.7.1 appnope==0.1.3 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 astroid==3.1.0 asttokens==2.4.1 async-lru==2.0.4 attrs==23.2.0 Babel==2.14.0 beautifulsoup4==4.12.3 black==23.12.1 bleach==6.1.0 boto3==1.34.30 botocore==1.34.30 bytecode==0.15.1 cattrs==23.2.3 certifi==2023.11.17 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==3.0.0 colorclass==2.2.2 comm==0.2.1 contourpy==1.2.0 coverage==7.4.1 cryptography==42.0.1 cycler==0.12.1 ddsketch==3.0.1 ddtrace==2.9.2 debugpy==1.8.0 decopatch==1.4.10 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 dicttoxml==1.7.16 dill==0.3.6 docopt==0.6.2 ecdsa==0.18.0 envier==0.5.1 exceptiongroup==1.2.0 executing==2.0.1 fastapi==0.110.3 fastjsonschema==2.19.1 fonttools==4.47.2 fqdn==1.5.1 github3.py==4.0.1 h11==0.14.0 httpcore==1.0.2 httptools==0.6.1 httpx==0.27.0 idna==3.6 importlib_metadata==7.1.0 iniconfig==2.0.0 ipykernel==6.29.0 ipython==8.20.0 ipywidgets==8.1.1 isoduration==20.11.0 isort==5.13.2 jedi==0.19.1 Jinja2==3.1.3 jmespath==1.0.1 joblib==1.3.2 json5==0.9.14 jsonlines==3.1.0 jsonpickle==3.0.2 jsonpointer==2.4 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-events==0.9.0 jupyter-lsp==2.2.2 jupyter_client==8.6.0 jupyter_core==5.7.1 jupyter_server==2.12.5 jupyter_server_terminals==0.5.2 jupyterlab==4.0.12 jupyterlab-widgets==3.0.9 jupyterlab_pygments==0.3.0 jupyterlab_server==2.25.2 kiwisolver==1.4.5 lab==8.0 llvmlite==0.39.1 lxml==4.6.5 makefun==1.15.2 MarkupSafe==2.1.4 matplotlib==3.8.2 matplotlib-inline==0.1.6 mccabe==0.7.0 mistune==3.0.2 mock==5.1.0 multi-key-dict==2.0.3 mypy-extensions==1.0.0 nbclient==0.9.0 nbconvert==7.14.2 nbformat==5.9.2 nest-asyncio==1.6.0 notebook==7.0.7 notebook_shim==0.2.3 numba==0.56.4 numpy==1.23.5 opentelemetry-api==1.25.0 overrides==7.7.0 packaging==23.2 pandas==2.0.2 pandocfilters==1.5.1 parso==0.8.3 pathspec==0.12.1 pbr==6.0.0 pexpect==4.9.0 pillow==10.2.0 pip-upgrader==1.4.15 platformdirs==2.6.2 pluggy==1.4.0 prometheus-client==0.19.0 prompt-toolkit==3.0.43 protobuf==5.27.1 psutil==5.9.8 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.5.1 pycparser==2.21 pydantic==2.7.4 pydantic-settings==2.3.3 pydantic_core==2.18.4 Pygments==2.17.2 PyJWT==2.8.0 pylint==3.1.0 pylint-plugin-utils==0.8.2 pylint-pydantic==0.3.2 pyparsing==3.1.1 pytest==7.4.4 pytest-asyncio==0.23.6 pytest-cases==3.8.5 pytest-cov==5.0.0 pytest-mock==3.14.0 python-dateutil==2.8.2 python-dotenv==1.0.1 python-jenkins==1.8.2 python-jose==3.3.0 python-json-logger==2.0.7 pytz==2023.4 PyYAML==6.0.1 pyzmq==25.1.2 qtconsole==5.5.1 QtPy==2.4.1 referencing==0.33.0 requests==2.31.0 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rpds-py==0.17.1 rsa==4.9 s3transfer==0.10.0 scikit-learn==1.2.2 scipy==1.12.0 Send2Trash==1.8.2 shap==0.45.0 simplejson==3.19.2 six==1.16.0 slicer==0.0.7 sniffio==1.3.0 soupsieve==2.5 stack-data==0.6.3 starlette==0.37.2 syrupy==4.6.1 tenacity==8.4.1 terminado==0.18.0 terminaltables==3.1.10 threadpoolctl==3.2.0 tinycss2==1.2.1 tokenize-rt==5.2.0 tomli==2.0.1 tomlkit==0.12.3 tornado==6.4 tqdm==4.66.1 traitlets==5.14.1 txt2tags==3.9 types-python-dateutil==2.8.19.20240106 typing_extensions==4.12.2 tzdata==2023.4 uri-template==1.3.0 uritemplate==4.1.1 urllib3==2.0.7 uvicorn==0.30.1 uvloop==0.19.0 watchfiles==0.21.0 wcwidth==0.2.13 webcolors==1.13 webencodings==0.5.1 websocket-client==1.7.0 websockets==12.0 widgetsnbextension==4.0.9 wrapt==1.16.0 xgboost==1.7.5 xmltodict==0.13.0 zipp==3.19.2How can we reproduce your problem?
$ddtrace-run python /opt/srv/runner.py
What is the result that you get?
What is the result that you expected?
ddtrace-run successfully initializes and runs the FastAPI app