DataDog / dd-trace-py

Datadog Python APM Client
https://ddtrace.readthedocs.io/
Other
546 stars 411 forks source link

In-flight spans are not flushed to Datadog on TERM signal #8252

Open bkempe opened 8 months ago

bkempe commented 8 months ago

Summary of problem

In-flight spans are not flushed to Datadog on TERM signal. This leaves out critical information, e.g. when a container gets terminated.

Which version of dd-trace-py are you using?

2.5.2

Which version of pip are you using?

23.2.1

How can we reproduce your problem?

test.py

import time

from ddtrace import tracer
import logging

logging.basicConfig(level=logging.DEBUG)

@tracer.wrap(name="test")
def test():
    time.sleep(60)
    return

test()
 ddtrace-run python test.py

then

pkill -f test.py

What is the result that you get?

No spans are sent to Datadog

What is the result that you expected?

Last span sent to Datadog via flush on TERM signal.

Mostly likely, the problem appeared here: https://github.com/DataDog/dd-trace-py/pull/7578#issuecomment-1919889866 which was a fix for https://github.com/DataDog/dd-trace-py/issues/7273

Therefore, it's unclear whether merely adding

register_on_exit_signal(tracer._atexit)

is safe.

emmettbutler commented 8 months ago

Thanks for pointing this out, @bkempe. The fix wasn't intended to change any existing functionality. We'll look into it.

bkempe commented 5 months ago

@emmettbutler @mabdinur This seems like a rather important issue and appears to have been abandoned?

emmettbutler commented 5 months ago

@bkempe hello! This issue hasn't been abandoned - it's still in our backlog of problems to look into. It's definitely an important issue.

bkempe commented 1 month ago

@emmettbutler Pinging again, since it's been ~7 months since the original issue has been reported.