DataDog / helm-charts

Helm charts for Datadog products
Apache License 2.0
330 stars 1.01k forks source link

Allow setting `logs_config.use_http` in agent chart #1362

Open jan-ludvik opened 3 months ago

jan-ludvik commented 3 months ago

Describe what happened: We are trialing Datadog NPM. Some findings are quite interesting - I noticed we are sending a lot of data to agent-intake.logs.datadoghq.com which apparently is for TCP transport. I also found this in agent status:

==========
Logs Agent
==========

    Reliable: Sending uncompressed logs in SSL encrypted TCP to agent-intake.logs.datadoghq.com on port 10516

    You are currently sending Logs to Datadog through TCP (either because logs_config.force_use_tcp or logs_config.socks5_proxy_address is set or the HTTP connectivity test has failed). To benefit from increased reliability and better network performances, we strongly encourage switching over to compressed HTTPS which is now the default protocol.

I have some suspicion this happens because when Datadog agent starts, the HTTPS check will fail - described here: https://docs.datadoghq.com/agent/logs/log_transport/?tab=https And the agent falls back to TCP. As HTTPS is recommended I would like to use that. The main reason is that I configured VPC Privatelinks from our VPCs to Datadog and this DNS name for TCP transport is not one o the endpoints. Therefore we send this through NAT which adds cost.

I would like to either:

My ultimate goal is to send logs over privatelink but if Datadog doesn't offer that Privatelink service, I would like to make sure we use HTTPS and endpoint that Datadog does offer. And HTTPS is recommended anyways.

Note: I restarted daemonset so all existing nodes get new datadog agent. It looked like this really worked and traffic to agent-intake.logs.datadoghq.com almost disappeared/went to 0.

Describe what you expected: Agents communicating over AWS Privatelink and not failing the HTTPS check. I don't see however an option to force HTTPS in the chart.

Steps to reproduce the issue: Try setting

logs_config:
  use_http: true

Additional environment details (Operating System, Cloud provider, etc): AWS EKS 1.25 with VPC CNI Datadog agent 7.51.1 Datadog helm chart 3.57.3

jan-ludvik commented 3 months ago

I found I can set this with env variables: agents.containers.agent.env or agents.containers.agent.envDict