search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
512 stars 404 forks source link

ddtrace loader fails execute modules in FastAPI app served by Uvicorn #9782

Open ztgzest opened 1 month ago

ztgzest commented 1 month ago

Summary of problem

ddtrace-run fails to run a FastAPI app served by uvicorn

It's specifically failing to load and execute modules needed by the app. When running the app without ddtrace-run (python /opt/srv/runner.py), it succeeds

Which version of dd-trace-py are you using?

2.9.2

Which version of pip are you using?

24.2.1

Which libraries and their versions are you using?

`pip freeze` annotated-types==0.6.0 anyio==3.7.1 appnope==0.1.3 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 astroid==3.1.0 asttokens==2.4.1 async-lru==2.0.4 attrs==23.2.0 Babel==2.14.0 beautifulsoup4==4.12.3 black==23.12.1 bleach==6.1.0 boto3==1.34.30 botocore==1.34.30 bytecode==0.15.1 cattrs==23.2.3 certifi==2023.11.17 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==3.0.0 colorclass==2.2.2 comm==0.2.1 contourpy==1.2.0 coverage==7.4.1 cryptography==42.0.1 cycler==0.12.1 ddsketch==3.0.1 ddtrace==2.9.2 debugpy==1.8.0 decopatch==1.4.10 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 dicttoxml==1.7.16 dill==0.3.6 docopt==0.6.2 ecdsa==0.18.0 envier==0.5.1 exceptiongroup==1.2.0 executing==2.0.1 fastapi==0.110.3 fastjsonschema==2.19.1 fonttools==4.47.2 fqdn==1.5.1 github3.py==4.0.1 h11==0.14.0 httpcore==1.0.2 httptools==0.6.1 httpx==0.27.0 idna==3.6 importlib_metadata==7.1.0 iniconfig==2.0.0 ipykernel==6.29.0 ipython==8.20.0 ipywidgets==8.1.1 isoduration==20.11.0 isort==5.13.2 jedi==0.19.1 Jinja2==3.1.3 jmespath==1.0.1 joblib==1.3.2 json5==0.9.14 jsonlines==3.1.0 jsonpickle==3.0.2 jsonpointer==2.4 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-events==0.9.0 jupyter-lsp==2.2.2 jupyter_client==8.6.0 jupyter_core==5.7.1 jupyter_server==2.12.5 jupyter_server_terminals==0.5.2 jupyterlab==4.0.12 jupyterlab-widgets==3.0.9 jupyterlab_pygments==0.3.0 jupyterlab_server==2.25.2 kiwisolver==1.4.5 lab==8.0 llvmlite==0.39.1 lxml==4.6.5 makefun==1.15.2 MarkupSafe==2.1.4 matplotlib==3.8.2 matplotlib-inline==0.1.6 mccabe==0.7.0 mistune==3.0.2 mock==5.1.0 multi-key-dict==2.0.3 mypy-extensions==1.0.0 nbclient==0.9.0 nbconvert==7.14.2 nbformat==5.9.2 nest-asyncio==1.6.0 notebook==7.0.7 notebook_shim==0.2.3 numba==0.56.4 numpy==1.23.5 opentelemetry-api==1.25.0 overrides==7.7.0 packaging==23.2 pandas==2.0.2 pandocfilters==1.5.1 parso==0.8.3 pathspec==0.12.1 pbr==6.0.0 pexpect==4.9.0 pillow==10.2.0 pip-upgrader==1.4.15 platformdirs==2.6.2 pluggy==1.4.0 prometheus-client==0.19.0 prompt-toolkit==3.0.43 protobuf==5.27.1 psutil==5.9.8 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.5.1 pycparser==2.21 pydantic==2.7.4 pydantic-settings==2.3.3 pydantic_core==2.18.4 Pygments==2.17.2 PyJWT==2.8.0 pylint==3.1.0 pylint-plugin-utils==0.8.2 pylint-pydantic==0.3.2 pyparsing==3.1.1 pytest==7.4.4 pytest-asyncio==0.23.6 pytest-cases==3.8.5 pytest-cov==5.0.0 pytest-mock==3.14.0 python-dateutil==2.8.2 python-dotenv==1.0.1 python-jenkins==1.8.2 python-jose==3.3.0 python-json-logger==2.0.7 pytz==2023.4 PyYAML==6.0.1 pyzmq==25.1.2 qtconsole==5.5.1 QtPy==2.4.1 referencing==0.33.0 requests==2.31.0 rfc3339-validator==0.1.4 rfc3986-validator==0.1.1 rpds-py==0.17.1 rsa==4.9 s3transfer==0.10.0 scikit-learn==1.2.2 scipy==1.12.0 Send2Trash==1.8.2 shap==0.45.0 simplejson==3.19.2 six==1.16.0 slicer==0.0.7 sniffio==1.3.0 soupsieve==2.5 stack-data==0.6.3 starlette==0.37.2 syrupy==4.6.1 tenacity==8.4.1 terminado==0.18.0 terminaltables==3.1.10 threadpoolctl==3.2.0 tinycss2==1.2.1 tokenize-rt==5.2.0 tomli==2.0.1 tomlkit==0.12.3 tornado==6.4 tqdm==4.66.1 traitlets==5.14.1 txt2tags==3.9 types-python-dateutil==2.8.19.20240106 typing_extensions==4.12.2 tzdata==2023.4 uri-template==1.3.0 uritemplate==4.1.1 urllib3==2.0.7 uvicorn==0.30.1 uvloop==0.19.0 watchfiles==0.21.0 wcwidth==0.2.13 webcolors==1.13 webencodings==0.5.1 websocket-client==1.7.0 websockets==12.0 widgetsnbextension==4.0.9 wrapt==1.16.0 xgboost==1.7.5 xmltodict==0.13.0 zipp==3.19.2

How can we reproduce your problem?

  1. Create a FastAPI app
  2. Write a py file that serves the app via Uvicorn
  3. Run py file using ddtrace: $ddtrace-run python /opt/srv/runner.py

What is the result that you get?

$ ddtrace-run python /opt/srv/runner.py

Traceback (most recent call last):

  File "/opt/srv/runner.py", line 32, in <module>

    uvicorn.run('my_module:create_app', factory=True, host=os.environ.get('API_HOST'), port=os.environ.get('API_PORT'), proxy_headers=True, forwarded_allow_ips='*', access_log=False)

  File "/usr/local/lib/python3.10/site-packages/uvicorn/main.py", line 577, in run

    server.run()

  File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run

    return asyncio.run(self.serve(sockets=sockets))

  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run

    return loop.run_until_complete(main)

  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete

  File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve

    await self._serve(sockets)

  File "/usr/local/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve

    config.load()

  File "/usr/local/lib/python3.10/site-packages/uvicorn/config.py", line 434, in load

    self.loaded_app = import_from_string(self.app)

  File "/usr/local/lib/python3.10/site-packages/uvicorn/importer.py", line 19, in import_from_string

    module = importlib.import_module(module_str)

  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "<frozen importlib._bootstrap_external>", line 883, in exec_module

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "/usr/local/lib/python3.10/site-packages/my_module/__init__.py", line 14, in <module>

    from .app import create_app

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/my_module/app.py", line 21, in <module>

    from model_deploy import DeployManager, config_dict

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/model_deploy/__init__.py", line 8, in <module>

    from model_deploy.utils import LOGGER_NAME, get_version

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/model_deploy/utils/__init__.py", line 2, in <module>

    from .utils import *

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/model_deploy/utils/utils.py", line 23, in <module>

    from pip._internal.operations import freeze

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/freeze.py", line 11, in <module>

    from pip._internal.req.constructors import (

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/__init__.py", line 9, in <module>

    from .req_install import InstallRequirement

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 40, in <module>

    from pip._internal.operations.install.wheel import install_wheel

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>

    from pip._vendor.distlib.scripts import ScriptMaker

  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked

  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/module.py", line 192, in _exec_module

    self.loader.exec_module(module)

  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distlib/scripts.py", line 66, in <module>

    for r in finder(distlib_package).iterator("")

  File "/usr/local/lib/python3.10/site-packages/pip/_vendor/distlib/resources.py", line 332, in finder

    raise DistlibException('Unable to locate finder for %r' % package)

pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'

What is the result that you expected?

ddtrace-run successfully initializes and runs the FastAPI app

ztgzest commented 1 month ago

I opened an issue in pip, too

emmettbutler commented 1 month ago

Thanks for the report, @ztgzest. We'll look into it.

alkio-netlight commented 4 weeks ago

We are facing a similar kind of problem when trying to run pip install while we have datadog agent (setup with https://s3.amazonaws.com/dd-agent/scripts/install_script_agent7.sh) enabled to monitor python processes. The error we get is the following

(.venv) sh-4.2$ pip3 install -r requirements.txt
Traceback (most recent call last):
  File "/app/.venv/bin/pip3", line 8, in <module>
    sys.exit(main())
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/cli/main.py", line 78, in main
    command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
    module = importlib.import_module(module_path)
  File "/usr/lib64/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 15, in <module>
    from pip._internal.cli.req_command import (
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 19, in <module>
    from pip._internal.index.package_finder import PackageFinder
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 31, in <module>
    from pip._internal.req import InstallRequirement
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 9, in <module>
    from .req_install import InstallRequirement
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 40, in <module>
    from pip._internal.operations.install.wheel import install_wheel
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/app/.venv/lib/python3.8/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>
    from pip._vendor.distlib.scripts import ScriptMaker
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "/opt/datadog-packages/datadog-apm-library-python/2.9.2/ddtrace_pkgs/site-packages-ddtrace-py3.8-manylinux2014/ddtrace/internal/module.py", line 220, in _exec_module
    self.loader.exec_module(module)
  File "/app/.venv/lib/python3.8/site-packages/pip/_vendor/distlib/scripts.py", line 66, in <module>
    for r in finder(distlib_package).iterator("")
  File "/app/.venv/lib/python3.8/site-packages/pip/_vendor/distlib/resources.py", line 332, in finder
    raise DistlibException('Unable to locate finder for %r' % package)
pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'

This started happening only recently and causing our services to fail.

alkio-netlight commented 3 days ago

Any updates on this?