search

DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
532 stars 408 forks source link

Argument 'filename' has incorrect type (expected str, got bytes) #6652

Closed dannosaur closed 12 months ago

dannosaur commented 1 year ago

Summary of problem

Argument 'filename' has incorrect type (expected str, got bytes) Unexpected error while exporting events. Please report this bug to https://github.com/DataDog/dd-trace-py/issues

Which version of dd-trace-py are you using?

1.17.3

Which version of pip are you using?

23.2.1

Which libraries and their versions are you using?

Only happens on Python 3.11, Python 3.10 doesn't exhibit this error.

`pip freeze` ably==2.0.0 amqp==5.1.1 annotated-types==0.5.0 anyio==3.7.1 arnparse==0.0.2 asgiref==3.7.2 async-timeout==4.0.2 attrs==23.1.0 Automat==22.10.0 balena-sdk==12.7.0 billiard==4.1.0 boto3==1.26.51 botocore==1.29.165 Brotli==1.0.9 bytecode==0.14.2 CacheControl==0.13.1 cachetools==5.3.1 cattrs==23.1.2 celery==5.3.1 certifi==2023.7.22 cffi==1.15.1 charset-normalizer==3.2.0 click==8.1.4 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.3.0 constantly==15.1.0 cronex==0.1.3.1 cryptography==41.0.2 cssselect2==0.7.0 datadog-api-client==2.14.0 ddsketch==2.0.4 ddtrace==1.17.3 defusedxml==0.7.1 Deprecated==1.2.14 dicttoxml2==2.0.0 diff-match-patch==20230430 Django==4.2.4 django-ace==1.19.0 django-admin-autocomplete-list-filter @ git+https://github.com/jzmiller1/django-admin-autocomplete-list-filter.git@239fca057b9aa29e92806fbaf2bb955f9fa8bedd django-appconf==1.0.5 django-axes==6.0.4 django-cleanup==7.0.0 django-constance==2.9.1 django-cors-headers==4.0.0 django-datadog-logger==0.6.1 django-encrypted-json-fields==1.0.4 django-extensions==3.2.3 django-filter==23.2 django-health-check==3.17.0 django-imagekit==4.1.0 django-import-export==3.2.0 django-ipware==5.0.0 django-json-widget==1.1.1 django-large-image==0.9.0 django-map-widgets==0.4.1 django-otp==1.2.1 django-picklefield==3.1 django-relativedelta==2.0.0 django-split-settings==1.0.1 django-storages==1.13.2 django-templated-mail==1.1.1 django-threadlocals==0.10 django-timezone-field==5.1 django-treebeard==4.7 django-vectortiles==0.2.0 djangorestframework==3.14.0 djangorestframework-camel-case==1.3.0 djangorestframework-gis==1.0 drf-nested-routers==0.91 drf-spectacular==0.24.2 drf-writable-nested==0.6.2 elasticsearch==7.13.4 elasticsearch-dsl==7.4.0 elementpath==4.1.4 envier==0.4.0 et-xmlfile==1.1.0 filelock==3.12.2 firebase-admin==6.2.0 fluent-logger==0.10.0 fonttools==4.40.0 fpdf2==2.5.0 future==0.18.3 GDAL==3.6.2 geographiclib==1.52 geopy==2.2.0 google-api-core==2.11.1 google-api-python-client==2.93.0 google-auth==2.15.0 google-auth-httplib2==0.1.0 google-cloud-core==2.3.3 google-cloud-firestore==2.11.1 google-cloud-storage==2.10.0 google-crc32c==1.1.2 google-resumable-media==2.5.0 googleapis-common-protos==1.59.1 googlemaps==4.10.0 grpcio==1.56.0 grpcio-status==1.49.1 gunicorn==20.1.0 h11==0.14.0 h2==4.1.0 hiredis==2.2.3 hpack==4.0.0 html5lib==1.1 httpcore==0.16.3 httplib2==0.22.0 httpx==0.23.3 humanize==4.1.0 hyperframe==6.0.1 hyperlink==21.0.0 idna==3.4 importlib-metadata==6.0.1 incremental==22.10.0 inflection==0.5.1 isodate==0.6.1 jmespath==1.0.1 JSON-log-formatter==0.5.2 jsonschema==4.18.2 jsonschema-specifications==2023.6.1 jwcrypto==1.4.2 kombu==5.3.1 large-image==1.19.3 large-image-source-gdal==1.19.3 Markdown==3.1.1 MarkupPy==1.14 mercantile==1.2.1 methoddispatch==3.0.2 msgpack==1.0.5 numpy==1.25.1 odfpy==1.4.1 openpyxl==3.1.2 opentelemetry-api==1.18.0 packaging==21.3 palettable==3.3.3 parameterized==0.8.1 phonenumbers==8.12.49 pilkit==2.0 Pillow==10.0.0 prompt-toolkit==3.0.39 proto-plus==1.22.3 protobuf==4.23.4 psutil==5.9.5 psycopg==3.1.9 psycopg-binary==3.1.9 pyasn1==0.5.0 pyasn1-modules==0.3.0 pycparser==2.21 pycurl==7.45.2 pydantic==2.1.1 pydantic_core==2.4.0 pydyf==0.7.0 pyee==9.1.1 pyhumps==3.5.3 PyJWT==2.6.0 pynamodb==5.1.0 pyOpenSSL==23.2.0 pyotp==2.8.0 pyparsing==2.4.7 pyphen==0.14.0 pyproj==3.6.0 pysaml2==7.4.2 python-dateutil==2.8.2 python-dotenv==1.0.0 python-json-logger==2.0.7 python-redis-lock==3.7.0 pytz==2021.1 PyYAML==6.0 qrcode==7.3.1 redis==4.5.4 referencing==0.29.1 requests==2.31.0 requests-aws4auth==1.0.1 rfc3986==1.5.0 rpds-py==0.8.10 rsa==4.9 s3transfer==0.6.1 semver==2.13.0 sentry-sdk==1.29.2 service-identity==23.1.0 simplejson==3.17.6 six==1.16.0 sniffio==1.3.0 sqlparse==0.4.4 tablib==3.5.0 tenacity==8.0.1 tinycss2==1.2.1 twilio==7.17.0 Twisted==22.10.0 typing_extensions==4.7.1 tzdata==2023.3 uritemplate==4.1.1 urllib3==1.26.16 uvicorn==0.22.0 vine==5.0.0 wcwidth==0.2.6 weasyprint==59.0 webencodings==0.5.1 websockets==10.4 wrapt==1.15.0 xlrd==2.0.1 xlwt==1.3.0 xmlschema==2.3.1 xmltodict==0.13.0 zipp==3.16.0 zope.interface==6.0 zopfli==0.2.2

How can we reproduce your problem?

Unclear. This error happens randomly and the stack trace doesn't give anything useful as to what caused the error in the first place.

TypeError: Argument 'filename' has incorrect type (expected str, got bytes)
  File "ddtrace/profiling/scheduler.py", line 53, in flush
    exp.export(events, start, self._last_export)
  File "ddtrace/profiling/exporter/http.py", line 175, in export
    profile, libs = super(PprofHTTPExporter, self).export(events, start_time_ns, end_time_ns)
  File "ddtrace/profiling/exporter/pprof.pyx", line 786, in ddtrace.profiling.exporter.pprof.PprofExporter.export
  File "ddtrace/profiling/exporter/pprof.pyx", line 313, in ddtrace.profiling.exporter.pprof._PprofConverter.convert_memalloc_heap_event
  File "ddtrace/profiling/exporter/pprof.pyx", line 238, in ddtrace.profiling.exporter.pprof._PprofConverter._to_locations
sanchda commented 1 year ago

Thanks for the report, @dannosaur!

I have a speculative fix, but I've been unable to reproduce this behavior on 3.11. It isn't 100% clear how this might arise. We've had other issues related to some as-of-yet unknown behavior when it comes to FrameObject subfields within the cpython runtime itself, but sadly we've only been able to protect against reported behavior rather than come to a deep understanding of the causal properties of those defects.

I'm wondering: would you be willing to run an experimental release of ddtrace in your environment? It would detect this situation and log some circumstantial information. This ask is independent of the fix, which I'll push as a PR tomorrow morning regardless. It'd serve to ensure the patch is indeed a comprehensive fix for the issue you're running into, and also hopefully give me some background information on how this might be happening.

If that's OK with you, I'd package what I have as a wheel and send it your way.

dannosaur commented 1 year ago

@sanchda absolutely, anything I can do to help. I can somewhat reliably reproduce this on our environment (and by reproduce, I mean I just leave our stack running and eventually the error shows up). Interestingly it hasn't raised the error for a couple days, I'm not sure what's changed, but we can definitely run an experimental version on our dev stack and see if we can get it to do it again.

sanchda commented 1 year ago

@dannosaur, awesome--thank you!

Just to confirm, it looks like you're running 3.11.4 in the configuration where the defect arises. The underlying machines are x86_64? I'll go ahead and get that build ready.

dannosaur commented 1 year ago

@sanchda that is correct, yes :)

sanchda commented 1 year ago

@dannosaur thank you! Testing a few things, then I'll send a build your way. It'll be a wheelfile I'll host out of a different repo.

jzmiller1 commented 12 months ago

Not stale. Very relevant. Keeping an eye out for the PR merge and release...