fix(grpc.aio): resolve distributed tracing concurrency bug [backport 2.17]

github-actions[bot] commented 6 days ago

Backport eb5a8d7d767152a7bef74ae2983be2acbd503bd8 from #11415 to 2.17.

Previously the tracer accessed the grpc metadata (which contained distributing tracing headers) directly from the Server’s HandlerCallDetails (here). This was not a safe operation, since the server’s handlerdetails is modified/set by every request. When multiple requests were handled concurrently the grpcaio integration would overwrite distributing tracing headers and clobber multiple unrelated trace chunks into one trace.

This PR ensures distributed tracing headers are extracted from the call handler details that's unique to the request: ServicerContext.

Checklist

[x] PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

[x] Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

datadog-dd-trace-py-rkomorn[bot] commented 6 days ago

Datadog Report

Branch report: backport-11415-to-2.17 Commit report: 259c3d4 Test service: dd-trace-py

:white_check_mark: 0 Failed, 592 Passed, 604 Skipped, 19m 41.71s Total duration (15m 46.66s time saved)

pr-commenter[bot] commented 6 days ago

Benchmarks

Benchmark execution time: 2024-11-26 03:33:00

Comparing candidate commit 259c3d43ce659c3e6ebd26ea0d92a3a654ae5efe in PR branch backport-11415-to-2.17 with baseline commit ea648a22dc880dba39e66fda6864a660fcbed412 in branch 2.17.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 388 metrics, 2 unstable metrics.

erikayasuda commented 4 days ago

/merge

dd-devflow[bot] commented 4 days ago

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-11-21 22:32:11 UTC :information_source: MergeQueue: pull request added to the queue

The median merge time in 2.17 is 0s.

2024-11-22 00:32:51 UTC :x: MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build took longer than expected. The current limit for the base branch '2.17' is 120 minutes.

Possible reasons:

target branch of PR is restricted to only allow up-to-date branches, but the pr is now outdated

github-actions[bot] commented 4 days ago

CODEOWNERS have been resolved as:

releasenotes/notes/grpc-use-ServicerContext-context-to-get-metadata-84c144a247267893.yaml  @DataDog/apm-python
ddtrace/contrib/internal/grpc/aio_server_interceptor.py                 @DataDog/apm-core-python @DataDog/apm-idm-python

wconti27 commented 3 days ago

/merge

dd-devflow[bot] commented 3 days ago

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-11-22 18:38:29 UTC :information_source: MergeQueue: pull request added to the queue

The median merge time in 2.17 is 0s.

2024-11-22 20:39:17 UTC :x: MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build took longer than expected. The current limit for the base branch '2.17' is 120 minutes.

Possible reasons:

target branch of PR is restricted to only allow up-to-date branches, but the pr is now outdated

mabdinur commented 18 hours ago

/merge

dd-devflow[bot] commented 18 hours ago

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-11-25 23:12:08 UTC :information_source: MergeQueue: waiting for PR to be ready

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals. Note: if you pushed new commits since the last approval, you may need additional approval. You can remove it from the waiting list with /remove command.

2024-11-25 23:21:18 UTC :information_source: MergeQueue: merge request added to the queue

The median merge time in 2.17 is 0s.

2024-11-26 01:21:38 UTC :x: MergeQueue: The build pipeline has timeout

The merge request has been interrupted because the build took longer than expected. The current limit for the base branch '2.17' is 120 minutes.

Possible reasons:

target branch of PR is restricted to only allow up-to-date branches, but the pr is now outdated

mabdinur commented 16 hours ago

/merge

dd-devflow[bot] commented 16 hours ago

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-11-26 01:47:52 UTC :information_source: MergeQueue: pull request added to the queue

The median merge time in 2.17 is 0s.

2024-11-26 02:51:21 UTC :x: MergeQueue: This merge request was updated

This PR is rejected because it was updated

DataDog / dd-trace-py