thread count keeps growing using ddtrace==2.14.2

DataDog / dd-trace-py

Datadog Python APM Client

https://ddtrace.readthedocs.io/

Other

548 stars 410 forks source link

thread count keeps growing using ddtrace==2.14.2 #11006

Open daillouf opened 2 weeks ago

daillouf commented 2 weeks ago

Hello everyone,

we are using ddtrace in production on our celery workers, and those seem to be creating extra threads

we had the issue when we initially upgraded all our unpinned dependencies, so we rolled back and I tested it one by one, the celery version is 5.3.6, and we also use celery-beats, and our worker pool should be multiprocessing.

our python version is 3.11 by the way,

let me know if i can give additional details or if you can’t reproduce,

Regards

sanchda commented 2 weeks ago

👋 thank you for the report!

Can you confirm the ddtrace version which is causing issues?

Also, I can't quite see the top of the dashboard, but it looks like this is a view of Docker Threads--is that correct? Would you be willing to share whether there is also an increase in the total number of processes (PIDs) in the container? This might help us narrow what category of behaviors to look at.

inikolaev commented 5 days ago

Not sure if related, but after upgrading to Python 3.12 and ddtrace==2.14.2 with profiling enabled I started to observe increased CPU utilization and profiler has been showing thousands of exceptions being thrown.

After I disabled profiling the CPU consumption has decreased:

Thrown exceptions during same time frame: