TypeError: expected str, bytes or os.PathLike object, not Path

denniszag commented 1 year ago

Summary of problem

When running ddtrace with latest version 1.14.0, after launching the app (using Profiler(...).start), I get the following error: TypeError: expected str, bytes or os.PathLike object, not Path

Notice that it does not happen on ddtrace==1.13.4

Stack trace:

TypeError: expected str, bytes or os.PathLike object, not Path
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/saved_packages/ddtrace/profiling/exporter/_packages.py", line 100, in <module>
    _FILE_PACKAGE_MAPPING = _build_package_file_mapping()
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/saved_packages/ddtrace/profiling/exporter/_packages.py", line 89, in _build_package_file_mapping
    mapping[fspath(f.locate())] = d
            ^^^^^^^^^^^^^^^^^^
TypeError: expected str, bytes or os.PathLike object, not Path

Which version of dd-trace-py are you using?

1.14.0

Which version of pip are you using?

pip 22.3.1 from /usr/local/lib/python3.11/packages.zip/pip (python 3.11)

Which libraries and their versions are you using?

`pip freeze`

> will be provided if needed

How can we reproduce your problem?

Run it on Python 3.11, on Kubernetes with Datadog agent gcr.io/datadoghq/agent:7.42.1 with the following mount:

    - name: datadog-apm-socket
      hostPath:
        path: /var/run/datadog/apm.socket
        type: Socket

P403n1x87 commented 1 year ago

@denniszag thanks for reporting this. This exception should be handled so it is unlikely to cause harm to your application. To mitigate this you may set DD_PROFILING_ENABLE_CODE_PROVENANCE=0 in your environment.

May I get some more information about your setup, please? In particular we would be interested in

pip freeze
the method used to start the application: is the ddtrace-run command being used?
the method used to start the profiler

I see a mention to Profiler(...).start and I was wondering why this method of starting the profiler is being used. Is there something forcing you to use this programmatic API over setting DD_PROFILING_ENABLED=1 with ddtrace-run? Is Profiler(...).start called as the very first thing during start up?

denniszag commented 1 year ago

Hi @P403n1x87. I would be happy to provide you with the information you asked for.

`pip freeze`

``` aenum==3.1.12 aiohttp==3.8.4 aiosignal==1.3.1 amqp==5.1.1 aniso8601==9.0.1 asn1crypto==1.5.1 astroid==2.15.5 async-generator==1.10 async-timeout==4.0.2 attrs==23.1.0 Authlib==1.2.0 azure-common==1.1.28 azure-core==1.27.0 azure-devops @ git+https://github.com/microsoft/azure-devops-python-api.git@b2d0009b2cc457452bc1f8c79b134de877680b06#subdirectory=azure-devops azure-identity==1.13.0 azure-keyvault==4.2.0 azure-keyvault-certificates==4.7.0 azure-keyvault-keys==4.8.0 azure-keyvault-secrets==4.7.0 azure-mgmt-authorization==3.0.0 azure-mgmt-core==1.4.0 azure-mgmt-resource==22.0.0 bandit==1.7.5 bcrypt==4.0.1 beautifulsoup4==4.12.2 blinker==1.6.2 boto3==1.26.147 botocore==1.29.147 Brotli==1.0.9 bytecode==0.14.2 cachetools==5.3.1 cattrs==23.1.2 cbor2==5.4.6 certifi==2023.5.7 cffi==1.15.1 cfgv==3.3.1 charset-normalizer==2.1.1 click==8.1.3 client-python @ git+https://github.com/rancher/client-python.git@f617457722f95eded2a74b301ab08d79502a5c3f colorlog==6.7.0 contextlib2==21.6.0 coverage==7.2.7 cryptography==39.0.2 dacite==1.8.1 datadog-api-client==2.13.2 ddsketch==2.0.4 ddtrace==1.13.4 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 dict2xml==1.7.3 dill==0.3.6 distlib==0.3.6 dnspython==2.3.0 dynaconf==3.1.12 ecdsa==0.18.0 envier==0.4.0 exceptiongroup==1.1.1 filelock==3.12.0 flake8==6.0.0 flake8-quotes==3.3.2 Flask==2.3.2 Flask-RESTful==0.3.10 flatdict==4.0.1 frozenlist==1.3.3 gevent==22.10.2 gitdb==4.0.10 GitPython==3.1.31 google-api-core==2.11.0 google-api-python-client==2.88.0 google-auth==2.19.1 google-auth-httplib2==0.1.0 google-auth-oauthlib==1.0.0 google-cloud-secret-manager==2.16.1 googleapis-common-protos==1.59.0 greenlet==2.0.2 grpc-google-iam-v1==0.12.6 grpcio==1.54.2 grpcio-status==1.54.2 h11==0.14.0 h2==4.1.0 hpack==4.0.0 httplib2==0.22.0 hyperframe==6.0.1 identify==2.5.24 idna==3.4 importlib-metadata==6.0.1 iniconfig==2.0.0 isodate==0.6.1 isort==5.12.0 itsdangerous==2.1.2 Jinja2==3.1.2 jmespath==1.0.1 jsonschema==4.17.3 kafka-python==2.0.2 kaitaistruct==0.10 kombu==5.3.0 kubernetes==26.1.0 lazy-object-proxy==1.9.0 ldap3==2.9.1 lxml==4.9.2 markdown-it-py==2.2.0 MarkupSafe==2.1.3 mccabe==0.7.0 mdurl==0.1.2 mock==5.0.2 msal==1.22.0 msal-extensions==1.0.0 msrest==0.7.1 multidict==6.0.4 mypy==1.3.0 mypy-boto3-iam==1.26.97 mypy-boto3-identitystore==1.26.107 mypy-extensions==1.0.0 nodeenv==1.8.0 oauth2==1.9.0.post1 oauth2client==4.1.3 oauthlib==3.2.2 observable==0.3.2 okta==2.9.2 opentelemetry-api==1.18.0 ordered-set==4.1.0 oscrypto==1.3.0 outcome==1.2.0 packaging==23.1 parameterized==0.8.1 paramiko==3.2.0 pbr==5.11.1 platformdirs==3.5.1 pluggy==1.0.0 policy-sentry==0.12.6 portalocker==2.7.0 pre-commit==3.3.2 proto-plus==1.22.2 protobuf==4.23.2 psycopg2-binary==2.9.6 pubnub==4.8.1 py==1.11.0 pyasn1==0.5.0 pyasn1-modules==0.3.0 PyBambooHR==0.8.1 pycodestyle==2.10.0 pycparser==2.21 pycryptodome==3.18.0 pycryptodomex==3.18.0 pydantic==1.10.8 pydash==7.0.4 pyflakes==3.0.1 PyGithub==1.58.2 Pygments==2.15.1 PyJWT==2.7.0 pylint==2.17.4 pymongo==4.3.3 PyMySQL==1.0.3 PyNaCl==1.5.0 pyodbc==4.0.39 pyOpenSSL==23.2.0 pyparsing==3.0.9 pyrsistent==0.19.3 PySocks==1.7.1 pytest==7.3.1 python-dateutil==2.8.2 python-dotenv==1.0.0 python-gitlab==3.14.0 python-jose==3.3.0 python-json-logger==2.0.7 python-sonarqube-api==2.0.3 pytz==2023.3 PyYAML==6.0 requests==2.31.0 requests-file==1.5.1 requests-mock==1.10.0 requests-oauthlib==1.3.1 requests-toolbelt==1.0.0 retry==0.9.2 rich==13.4.1 ringcentral==0.7.18 rsa==4.9 s3transfer==0.6.1 schema==0.7.5 selenium==4.9.1 selenium-wire==5.1.0 shortuuid==1.0.11 simple-salesforce==1.12.4 six==1.16.0 slack-sdk==3.21.3 smmap==5.0.0 sniffio==1.3.0 snowflake-connector-python==3.0.3 sortedcontainers==2.4.0 soupsieve==2.4.1 spur==0.3.23 SQLAlchemy==1.4.46 SQLAlchemy-Utils==0.39.0 stevedore==5.1.0 tableauserverclient==0.25 tenacity==8.2.2 tomlkit==0.11.8 trio==0.22.0 trio-websocket==0.10.2 types-cachetools==5.3.0.5 types-Deprecated==1.2.9.2 types-mock==5.0.0.6 types-python-dateutil==2.8.19.13 types-PyYAML==6.0.12.10 types-requests==2.31.0.1 types-retry==0.9.9.3 types-urllib3==1.26.25.13 typing_extensions==4.6.3 uritemplate==4.1.1 urllib3==1.26.16 vine==5.0.0 virtualenv==20.23.0 websocket-client==1.5.2 Werkzeug==2.3.4 wrapt==1.15.0 wsproto==1.2.0 xmltodict==0.13.0 yarl==1.9.2 zeep==4.2.1 zipp==3.15.0 zope.event==4.6 zope.interface==6.0 zstandard==0.21.0 ```

The method used to start the application: is the ddtrace-run command being used? No. We start the profiler programatically.

The method used to start the profiler Snippet:

profiler = Profiler(env='service_version', service='service_name', version='service_version')
profiler.start()

We start the profiler programmatically because we have a flag in the configuration that allows us enabling/disabling the profiler easily (from remote environment). In addition we would like to omit changing the deployment method of the application, and avoid coupling with Datadog tracer at the Kubernetes/Docker level.

Let me know if you need any more details.

P403n1x87 commented 1 year ago

@denniszag thanks for the extra details. First of all, let me reiterate that the error you see should not be of concern. What it means is that the profiler is failing to build the mapping of sources to distributions, meaning that it won't be able to report code provenance information when that happens (on the UI level, this means that you won't be able to filter profiles for just your code, as opposed to stdlib/third-party code). Your application should not be impacted by this error. Code provenance is enabled by default on 1.14.0 but not on earlier versions. So one way to make this error disappear should be

import os
os.environ["DD_PROFILING_ENABLE_CODE_PROVENANCE"] = "false"

just before starting the profiler.

This said, I was not able to reproduce the issue so I would like to get some more details from you if that's ok. From your pip freeze I see gevent and flask. Am I right in thinking that we are talking about a Flask application running with gevent workers? What command do you use to start the application. Is there any other interaction with the ddtrace library, apart from programmatically starting the profiler? Our docs recommend importing ddtrace.auto as early as possible, and before gevent monkey-patching to ensure that the library works as expected when ddtrace-run is not being used. Is ddtrace.auto being imported at any point during the start up of your application? If possible, I'd recommend initialising ddtrace inside a sitecustomize.py script, e.g.

# in custom sitecustomize.py
import ddtrace.auto. # noqa

from ddtrace.profiling import Profiler

profiler = Profiler(env='service_version', service='service_name', version='service_version')
profiler.start()

This is to ensure that the import of ddtrace.auto happens before gevent monkey-patching.

denniszag commented 1 year ago

Hi @P403n1x87, I would be happy to provide you with more details (but some of them maybe should be in a private conversation).

Actually, the application isn't a flask running with gevent. I think that those dependencies were there because they are dev-only dependencies (I run it on my local computer).

Regarding the monkey patching - it is possible that the import doesn't run as early as possible, especially because if requires to load the settings and more configuration related modules. But, it is one of the first imports. Notice that the import is a local import, that occurs only if the tracer is enabled in remote configuration.

Regarding ddtrace.auto - No. It is not imported in the app, and the only Datadog-related import we have is from ddtrace.profiling import Profiler.

emmettbutler commented 10 months ago

Manually doing what the Github Action was supposed to do (fixed in https://github.com/DataDog/dd-trace-py/pull/7835):

This issue has been automatically closed after six months of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.

DataDog / dd-trace-py