Prevent duplicate DT headers for grpc

[kanderson250]

Description

We got a report from NR-231941 that distributed tracing was broken between two apps using grpc. Specifically, traceIds from the upstream app were not being used by the downstream app, and the two apps were not being linked in the UI.

After investigation we discovered that the distributed tracing headers newrelic, traceparent, and tracestate were being duplicated (ie same key, same value) in the upstream app's outbound headers. Because the headers were duplicated, the DT payload was not accepted by the downstream app.

To do:

1. Prevent duplicate distributed tracing headers from being added to the metadata object in the OutboundHeadersWrapper class of the grpc instrumentation. This needs to be done for all versions of grpc.
2. In W3CTraceParentParser.parseHeaders we check that the traceparent header has size 1. Within this check, add additional logic that will validate the headers if their size is not 1 but their values agree. The supportability metric in that check should still be generated.
3. Add a WARNING level log message withinW3CTraceParentParser.parseHeaders when multiple trace parent headers are found.

   Considerations

Are there any valid cases for duplicate DT headers? Is overwriting the DT headers acceptable, or should we just prevent adding new duplicate headers?

Acceptance Criteria

• Outbound headers are not duplicated
• The issue is fixed for all versions of the grpc instrumentation
• New tests for duplicate headers are added
• All tests passing

[workato-integration[bot]]

https://new-relic.atlassian.net/browse/NR-233559

[kanderson250]

Note that the GTSE includes a repro app.