DataDog / dd-trace-go

Datadog Go Library including APM tracing, profiling, and security monitoring.
https://docs.datadoghq.com/tracing/
Other
651 stars 435 forks source link

ddtrace/tracer: null DD_SERVICE_MAPPING is overriding service name #1164

Open matlegit opened 2 years ago

matlegit commented 2 years ago

Hi there,

We ran into this issue in both v1.35 and v1.36. There is potentially a bug around which DataDog Service the agent reports data back to. In v1.34 we were sending APM data to my-service but after upgrading the tracer, it now sends to a Service called echo which looks to be the default set here. We're using Echo github.com/labstack/echo/v4 v4.6.3

The env var we're using in our containerized app:

        - name: DD_SERVICE
          value: my-service

Related PR: https://github.com/DataDog/dd-trace-go/pull/1077

Log:

{"time":"2022-02-08T23:50:10.416286813Z","level":"INFO","prefix":"echo","file":"tracer.go","line":"20","message":"Datadog Tracer v1.36.0 INFO: DATADOG TRACER CONFIGURATION {\"date\":\"2022-02-08T23:50:10Z\",\"os_name\":\"Linux (Unknown Distribution)\",\"os_version\":\"unknown\",\"version\":\"v1.36.0\",\"lang\":\"Go\",\"lang_version\":\"go1.16.3\",\"env\":\"dev\",\"service\":\"my-service\",\"agent_url\":\"http://10.32.10.79:8126/v0.4/traces\",\"agent_error\":\"\",\"debug\":false,\"analytics_enabled\":true,\"sample_rate\":\"NaN\",\"sampling_rules\":null,\"sampling_rules_error\":\"\",\"service_mappings\":null,\"tags\":{\"kube_namespace\":\"my-namespace\",\"notify_group\":\"unused\",\"runtime-id\":\"8bc3cf02-0a9d-4ea5-9bfe-2955f780259d\"},\"runtime_metrics_enabled\":false,\"health_metrics_enabled\":false,\"profiler_code_hotspots_enabled\":false,\"profiler_endpoints_enabled\":false,\"dd_version\":\"29c37dc0d21cbba8bf6908971fe8e2e919d424bc\",\"architecture\":\"amd64\",\"global_service\":\"my-service\",\"lambda_mode\":\"false\",\"appsec\":false,\"agent_features\":{\"DropP0s\":true,\"Stats\":true,\"StatsdPort\":0}}\n"}

Based on the official docs

DD_SERVICE_MAPPING
    Default: null
    Dynamically rename services through configuration. Services can be separated by commas or spaces, for example: mysql:mysql-service-name,postgres:postgres-service-name, mysql:mysql-service-name postgres:postgres-service-name.

Our interpretation is if the service mapping is null it shouldn't interfere with the service naming.

We were able to get around the bug by setting DD_SERVICE_MAPPING to rename the service to its original name

- name:  DD_SERVICE_MAPPING
  value: "echo:my-service"

Log:

{"time":"2022-02-08T23:53:30.700977262Z","level":"INFO","prefix":"echo","file":"tracer.go","line":"20","message":"Datadog Tracer v1.36.0 INFO: DATADOG TRACER CONFIGURATION {\"date\":\"2022-02-08T23:53:30Z\",\"os_name\":\"Linux (Unknown Distribution)\",\"os_version\":\"unknown\",\"version\":\"v1.36.0\",\"lang\":\"Go\",\"lang_version\":\"go1.16.3\",\"env\":\"dev\",\"service\":\"my-service\",\"agent_url\":\"http://10.32.10.79:8126/v0.4/traces\",\"agent_error\":\"\",\"debug\":false,\"analytics_enabled\":true,\"sample_rate\":\"NaN\",\"sampling_rules\":null,\"sampling_rules_error\":\"\",\"service_mappings\":{\"echo\":\"my-service\"},\"tags\":{\"kube_namespace\":\"my-namespace\",\"notify_group\":\"unused\",\"runtime-id\":\"6c1f6283-1e29-4cf1-9b16-ebe672cc59f1\"},\"runtime_metrics_enabled\":false,\"health_metrics_enabled\":false,\"profiler_code_hotspots_enabled\":false,\"profiler_endpoints_enabled\":false,\"dd_version\":\"29c37dc0d21cbba8bf6908971fe8e2e919d424bc\",\"architecture\":\"amd64\",\"global_service\":\"my-service\",\"lambda_mode\":\"false\",\"appsec\":false,\"agent_features\":{\"DropP0s\":true,\"Stats\":true,\"StatsdPort\":0}}\n"}

Hope this helps, thanks

dianashevchenko commented 2 years ago

@matlegit from what I see, null service mapping can't interfere with the service naming, so I couldn't reproduce. If the problem still persists, please feel free to share more info / snippets 🙌