ddtrace==2.13.1 profiler issue

[doudoujay]

After i updated our docker's ddtrace from ddtrace==2.9.3 to ddtrace==2.13.1, we observed some weird behaviors on profiler. We upgraded on Oct 11.

Heap - Bytes in Use drop to 0

Allocated Bytes Significantly increased - Huge chunk of memory consumed by periodic ensure_binary

Increased CPU Time

[doudoujay]

any updates?

[doudoujay]

@mabdinur @taegyunkim any updates on this?

[taegyunkim]

@doudoujay Hi could you share environment variables that you're giving to ddtrace? e.g. DD_PROFILING_* and DD_TRACE_*. Could you also share Python version too?

[doudoujay]

ENV DD_LOGS_ENABLED=true
ENV DD_LOGS_INJECTION=true
ENV DD_TRACE_ENABLED=true
ENV DD_PROFILING_ENABLED=true
# https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/python/otel/
ENV DD_TRACE_OTEL_ENABLED=true
# https://docs.datadoghq.com/profiler/connect_traces_and_profiles/#identify-code-hotspots-in-slow-traces
# only available for ver 2.10.5+
ENV DD_PROFILING_TIMELINE_ENABLED=true
# https://docs.datadoghq.com/dynamic_instrumentation/enabling/python/?tab=environmentvariables
ENV DD_DYNAMIC_INSTRUMENTATION_ENABLED=true
# https://docs.datadoghq.com/tracing/metrics/runtime_metrics/python/
ENV DD_RUNTIME_METRICS_ENABLED=true
# https://docs.datadoghq.com/infrastructure/process/?tab=docker
ENV DD_PROCESS_CONFIG_PROCESS_COLLECTION_ENABLED=true
# code tracking
ARG DD_GIT_REPOSITORY_URL
ARG DD_GIT_COMMIT_SHA
ENV DD_GIT_REPOSITORY_URL=${DD_GIT_REPOSITORY_URL}
ENV DD_GIT_COMMIT_SHA=${DD_GIT_COMMIT_SHA}

@taegyunkim

[taegyunkim]

1. ensure_binary_or_empty shows up in your profiles as you've set DD_PROFILING_TIMELINE_ENABLED. The flag enables native profile exporter which uses ensure_binary_or_empty to convert Python str to utf-8 encoded bytes which is then passed to C++ native code as std::string_view.
2. You don't see any frame below ensure_binary_or_empty as the function calls CPython's built-in str.encode() which is implemented in C here and dd-trace-py profiler doesn't show native frames.
3. Looks like str.encode() always creates an object, allocating memory, even if the given str object is already in the desired encoding, in our case utf-8. We probably should use lower level CPython API, probably PyUnicode_AsUTF8AndSize(), which simply returns const char*, and pass that to C++ native code.

Thank you for pointing this out @doudoujay, will soon work on a fix for this, < 2 weeks.

[doudoujay]

So i guess for short term solution would be we turn off DD_PROFILING_TIMELINE_ENABLED? @taegyunkim

[taegyunkim]

Yes, could you try turning it off?

[doudoujay]

@taegyunkim hey its still taking significant chunk of CPU cycle and memory - dd version 2.14.4

dd flags:

ENV DD_LOGS_ENABLED=true
ENV DD_LOGS_INJECTION=true
ENV DD_TRACE_ENABLED=true
ENV DD_PROFILING_ENABLED=true
# https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/python/otel/
ENV DD_TRACE_OTEL_ENABLED=true
# https://docs.datadoghq.com/dynamic_instrumentation/enabling/python/?tab=environmentvariables
ENV DD_DYNAMIC_INSTRUMENTATION_ENABLED=true
# https://docs.datadoghq.com/tracing/metrics/runtime_metrics/python/
ENV DD_RUNTIME_METRICS_ENABLED=true
# https://docs.datadoghq.com/infrastructure/process/?tab=docker
ENV DD_PROCESS_CONFIG_PROCESS_COLLECTION_ENABLED=true
# code tracking
ARG DD_GIT_REPOSITORY_URL
ARG DD_GIT_COMMIT_SHA
ENV DD_GIT_REPOSITORY_URL=${DD_GIT_REPOSITORY_URL}
ENV DD_GIT_COMMIT_SHA=${DD_GIT_COMMIT_SHA}