datadog agent consul autodiscovery integration config with acl token

[fizzamahdi]

Describe what happened: Datadog agent consul check returns 403 forbidden, even when ACL token is set in autodiscovery integration config. I've exec'd into the datadog agent container and manually run a curl command with the token header set and it is able to authenticate to Consul so I know the token has the necessary permissions.

Describe what you expected: datadog agent would be able to authenticate to consul using acl token provided in autodiscovery integration config

Steps to reproduce the issue:

• Deploy Consul agents (clients and server) to k8s cluster
• Deploy datadog agent to k8s cluster
• Add annotations to spec.template.metadata of Consul server statefulset and Consul client daemonset:

  annotations:
  ad.datadoghq.com/consul.check_names: '["consul"]'
  ad.datadoghq.com/consul.init_configs: '[{}]'
  ad.datadoghq.com/consul.instances: '[{"url": "http://%%host%%:8500","acl_token": "<my_super_permissive_bootstrap_token>"}]'

• after consul pods have restarted, restart the datadog agent pods

Additional environment details (Operating System, Cloud provider, etc):

• running all containers on AWS EKS
• Consul is setup to use ACLs
• tested using the bootstrap ACL token for datadog agent to authenticate since that token has access to everything

I could be missing something in my configuration, but didn't see anything specific in the docs. I saw this snippet in the example configuration file (etc/datadog-agent/conf.d/consul.d/conf.yaml.example) in the datadog agent and decided to set the ACL token because of that:

    ## @param acl_token - string - optional
    ## ACL token to use for authentication.
    #
    # acl_token: '<TOKEN>'

Another team manages the datadog agent in our clusters, so I told them another option for us is to configure consul to push data to the agent through DogStatsD instead of using datadog's autodiscovery, but before going that route I would like to figure out how to get the autodiscovery to work with ACLs.

Thank you in advance for any help!

[fizzamahdi]

here are the WARN and ERROR level logs that pertain to consul that I'm seeing from the datadog agent:

2020-07-01 20:23:08 UTC | CORE | WARN | (pkg/autodiscovery/autoconfig.go:528 in resolveTemplateForService) | error resolving template consul for service docker://2c8fc2195d6b9a2f279fda04be89e396d1e18f0df214e53e89129d3f439a1114: ignoring config from file:/etc/datadog-agent/conf.d/consul.d/auto_conf.yaml: another config is defined for the check consul
2020-07-01 20:23:09 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:116 in LogMessage) | consul:a77e58d6bbc1bb1d | (consul.py:87) | Consul request to http://100.64.94.25:8500/v1/agent/self failed
Traceback (most recent call last):
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/consul/consul.py", line 76, in consul_request
    resp.raise_for_status()
  File "/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://100.64.94.25:8500/v1/agent/self
2020-07-01 20:23:09 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check consul: [{"message": "403 Client Error: Forbidden for url: http://100.64.94.25:8500/v1/agent/self", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 820, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/consul/consul.py\", line 244, in check\n    self._collect_metadata()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/consul/consul.py\", line 528, in _collect_metadata\n    local_config = self._get_local_config()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/consul/consul.py\", line 103, in _get_local_config\n    self._local_config = self.consul_request('/v1/agent/self')\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/consul/consul.py\", line 76, in consul_request\n    resp.raise_for_status()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/models.py\", line 940, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://100.64.94.25:8500/v1/agent/self\n"}]