Open doudoujay opened 1 week ago
any updates?
@mabdinur @taegyunkim any updates on this?
@doudoujay Hi could you share environment variables that you're giving to ddtrace? e.g. DD_PROFILING_*
and DD_TRACE_*
. Could you also share Python version too?
ENV DD_LOGS_ENABLED=true
ENV DD_LOGS_INJECTION=true
ENV DD_TRACE_ENABLED=true
ENV DD_PROFILING_ENABLED=true
# https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/python/otel/
ENV DD_TRACE_OTEL_ENABLED=true
# https://docs.datadoghq.com/profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces
# only available for ver 2.10.5+
ENV DD_PROFILING_TIMELINE_ENABLED=true
# https://docs.datadoghq.com/dynamic_instrumentation/enabling/python/?tab=environmentvariables
ENV DD_DYNAMIC_INSTRUMENTATION_ENABLED=true
# https://docs.datadoghq.com/tracing/metrics/runtime_metrics/python/
ENV DD_RUNTIME_METRICS_ENABLED=true
# https://docs.datadoghq.com/infrastructure/process/?tab=docker
ENV DD_PROCESS_CONFIG_PROCESS_COLLECTION_ENABLED=true
# code tracking
ARG DD_GIT_REPOSITORY_URL
ARG DD_GIT_COMMIT_SHA
ENV DD_GIT_REPOSITORY_URL=${DD_GIT_REPOSITORY_URL}
ENV DD_GIT_COMMIT_SHA=${DD_GIT_COMMIT_SHA}
@taegyunkim
ensure_binary_or_empty
shows up in your profiles as you've set DD_PROFILING_TIMELINE_ENABLED
. The flag enables native profile exporter which uses ensure_binary_or_empty
to convert Python str
to utf-8 encoded bytes
which is then passed to C++ native code as std::string_view
. ensure_binary_or_empty
as the function calls CPython's built-in str.encode()
which is implemented in C here and dd-trace-py profiler doesn't show native frames.str.encode()
always creates an object, allocating memory, even if the given str object is already in the desired encoding, in our case utf-8
. We probably should use lower level CPython API, probably PyUnicode_AsUTF8AndSize()
, which simply returns const char*
, and pass that to C++ native code.Thank you for pointing this out @doudoujay, will soon work on a fix for this, < 2 weeks.
So i guess for short term solution would be we turn off DD_PROFILING_TIMELINE_ENABLED
? @taegyunkim
Yes, could you try turning it off?
@taegyunkim hey its still taking significant chunk of CPU cycle and memory - dd version 2.14.4
dd flags:
ENV DD_LOGS_ENABLED=true
ENV DD_LOGS_INJECTION=true
ENV DD_TRACE_ENABLED=true
ENV DD_PROFILING_ENABLED=true
# https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/python/otel/
ENV DD_TRACE_OTEL_ENABLED=true
# https://docs.datadoghq.com/dynamic_instrumentation/enabling/python/?tab=environmentvariables
ENV DD_DYNAMIC_INSTRUMENTATION_ENABLED=true
# https://docs.datadoghq.com/tracing/metrics/runtime_metrics/python/
ENV DD_RUNTIME_METRICS_ENABLED=true
# https://docs.datadoghq.com/infrastructure/process/?tab=docker
ENV DD_PROCESS_CONFIG_PROCESS_COLLECTION_ENABLED=true
# code tracking
ARG DD_GIT_REPOSITORY_URL
ARG DD_GIT_COMMIT_SHA
ENV DD_GIT_REPOSITORY_URL=${DD_GIT_REPOSITORY_URL}
ENV DD_GIT_COMMIT_SHA=${DD_GIT_COMMIT_SHA}
After i updated our docker's ddtrace from ddtrace==2.9.3 to ddtrace==2.13.1, we observed some weird behaviors on profiler. We upgraded on Oct 11.
Heap - Bytes in Use drop to 0
Allocated Bytes Significantly increased - Huge chunk of memory consumed by periodic
ensure_binary
Increased CPU Time