Tags from DD_TAGS are not added to service checks

[2rs2ts]

From agent status, here are the host tags

    host tags:
      bt_platform:blue
      datacenter:qa
      env:qa
      environment:qa
      role:kube-worker
      kubernetes
      blue_datacenter:qa
      blue_environment:qa

From kubectl describe pod, here is the env var:

      DD_TAGS:                           bt_platform:blue datacenter:qa env:qa environment:qa role:kube-worker kubernetes blue_datacenter:qa blue_environment:qa

Describe what happened: We have several service checks we add as YAML to /conf.d in our container. They show up in the UI just fine but are missing all these tags. The only way these tags show up is if we add them manually to the YAML definition. This occurs whether or not we define any additional tags in the YAML config.

Describe what you expected: These tags should get added to service checks just like they get added to metrics.

Steps to reproduce the issue: Deploy an agent (in our case, to K8s) with the DD_TAGS environment variable set. Add some service checks (in our case, http_check, tcp_check, prometheus, and more.) Watch the magic not happen.

Additional environment details (Operating System, Cloud provider, etc): Kubernetes 1.14.4. AWS. Debian stretch. Agent 6.13.0.

[Simwar]

Hi Andrew,

Thanks for reaching out! I see that you already opened a ticket for this feature request. Unfortunately, this request did not get a lot of traction since the ticket has been opened. Is there a specific use-case I can add to update the feature request? I will reach out to the Product team to see where we stand.

Thank you

Simon

[2rs2ts]

@Simwar yeah, we have 7-8 checks like this and having to template in these tags every time is annoying. What's more, some of these checks have to be templated at runtime (to get the node hostname in the config) so we have an initContainer for that and this limitation forces us to duplicate our list of tags in two places (prone to developer error) and forces us to provide extra arguments to the initContainer to get the right info in there (even more boilerplate.)

This behavior is not intuitive at all, I wouldn't even consider this a feature request. It's a bug. You don't have to do this for statsd metrics or for autodiscovery metrics, those just get the right tags.

I don't even remember filing a FR for this. Is this from a really old support ticket or something?

[Simwar]

Hi Andrew,

Yes, it is one year old and closed ever since. I have a follow-up question regarding what you mean by "show up in the UI". Do you mean, you don't see them in the service check summary (https://app.datadoghq.com/check/summary) or you cannot use them in the monitor UI?

Thanks

[2rs2ts]

I meant the service check summary.

[Simwar]

Hi Andrew,

Thanks for your answer! This is unfortunately a known bug in this UI. You can still look for these tags and the checks should be filtered by it, but the tags won't be displayed. If you want to be notified directly when the issue is fixed, you can open a ticket with our solutions team and mention this github issue. Otherwise, we will also make sure to close this github issue when this is fixed.

Thanks

[2rs2ts]

Hmm, no, if you try to filter for the tag any check that is supposed to have it won't show up. And I don't trust that alerting will work without the tags showing up.

[Simwar]

Hi Andrew,

Thanks for the feedback. If the query is having the correct tags in the monitor UI (when you create the monitor), the alerting should work correctly. Could you open a ticket with our solutions team so we can investigate further on this case?

Thanks

[2rs2ts]

That's too much of a pain and a risk for me to try to reproduce. As far as I'm concerned if the service check UI doesn't show the tags the tags aren't there.