Open sampritipanda opened 3 months ago
:wave: Thank you for the report! Unfortunately, this is a known problem in the "stack v2" implementation of the profiler on Python 3.12 (it does not occur in 3.11 or earlier). If you haven't tried the "legacy" stack collector (just omit the DD_PROFILING_STACK_V2_ENABLED
environment variable, or use DD_PROFILING_STACK_V2_ENABLED=false
), then please give it a shot and see if it offers you some relief.
If you're using "stack v2" for a reason (such as, avoiding the even greater number of segfaults originating from the cpython runtime for the legacy stack collector), then please ignore that advice. π
This may actually have been fixed in ddtrace 3.8.4. I'll be testing and working on a fix this upcoming week. I'll check back in on Math 15th or so to confirm whether that version actually has the fix. If you'd like to try the new release to see if it helps, please let me know how that goes!
Thanks, I indeed had DD_PROFILING_STACK_V2_ENABLED
and DD_PROFILING_EXPORT_LIBDD_ENABLED
enabled to mitigate segfaults from Python 3.11. Should I disable both of them or just STACK_V2?
:wave: sorry, I'm not sure why I lost track of this thread.
If you're using stack v2 in order to mitigate segfaults, then unfortunately there's not much relief. Working on a fix.
Summary of problem
I'm getting frequent segmentation faults in my application after I started to use Python 3.12
Which version of dd-trace-py are you using?
ddtrace = "^2.8.1"
How can we reproduce your problem?
Not very reproducible so far.
What is the result that you get?
Here's a stack trace of the segfault from one of the core dumps generated:
The docker container we use is:
thehale/python-poetry:1.8.2-py3.12-slim
It's somewhat unclear what causes this segfault, but all of our segfaults have this exact same stack trace. I can share some code pointers which each stack frame refers to that I found while trying to root cause this:
4 - (I believe this the python allocator malloc implementation) 5 - https://github.com/DataDog/dd-trace-py/blob/main/ddtrace/profiling/collector/_memalloc.c#L116 9 - https://github.com/python/cpython/blob/3.12/Objects/longobject.c#L542-L543 10 - https://github.com/P403n1x87/echion/blob/main/echion/strings.h#L104
What is the result that you expected?
No segfaults pls π Really like the product otherwise.