Log agent config defaults change when additional_endpoints are enabled

[agy]

Describe what happened:

When enabling additional_endpoints in the logs_config section, the logs transport change from using HTTP to TCP. This is unexpected since I thought that I was only changing a subordinate endpoint but this also had an effect on the primary endpoint's configuration.

When working around this issue but setting the transport manually to HTTPS, the additional endpoint does not use compression without also setting this value manually.

Describe what you expected:

When enabling an additional endpoint, I expect my "main" endpoint's transport to remain unchanged and for the additional endpoint's defaults to be consistent with the "main" endpoint's defaults.

If the main logs transport defaults to using HTTPS with compression, I expect additional logs endpoints to use these defaults as well.

Steps to reproduce the issue:

Note: After each edit of the config, the datadog-agent was restarted. This step and all the datadog-agent output has been omitted below for brevity.

Setup

Create a new Ubuntu 18.04 instance and install the datadog-agent by copy and pasting the instructions from the https://app.datadoghq.com/account/settings#ubuntu page.

Enable logs

This is my logs enabled configuration which works as expected. Logs are being sent compressed using the HTTPS transport.

logs_enabled: true
logs_config:
  dd_url: agent-http-intake.logs.datadoghq.com

Output from datadog-agent status

    Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443

Add an additional endpoint

Now the transport for both endpoints have been changed to TCP!?

logs_enabled: true
logs_config:
  dd_url: agent-http-intake.logs.datadoghq.com
  additional_endpoints:
    - api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      host: agent-http-intake.logs.datadoghq.com

Output from datadog-agent status

    Sending uncompressed logs in SSL encrypted TCP to agent-http-intake.logs.datadoghq.com on port 10516
    Additional: Sending uncompressed logs in SSL encrypted TCP to agent-http-intake.logs.datadoghq.com on port 0

    You are currently sending Logs to Datadog through TCP (either because logs_config.use_tcp or logs_config.socks5_proxy_address is set or the HTTP connectivity test has failed). To benefit from increased reliability and better network performances, we strongly encourage switching over to compressed HTTPS which is now the default protocol.

Work around the transport issue, by setting it manually

Now both transports are now using HTTPS but the additional one isn't using compression!?

logs_enabled: true
logs_config:
  use_http: true
  dd_url: agent-http-intake.logs.datadoghq.com
  additional_endpoints:
    - api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      host: agent-http-intake.logs.datadoghq.com

Output from datadog-agent status

    Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443
    Additional: Sending uncompressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443

Work around the transport and compression issues, by setting it manually

This sets the transport for all endpoints to HTTPS and uses compression.

logs_enabled: true
logs_config:
  use_http: true
  dd_url: agent-http-intake.logs.datadoghq.com
  additional_endpoints:
    - api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      host: agent-http-intake.logs.datadoghq.com
      use_compression: true

Output from datadog-agent status

    Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443
    Additional: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.com on port 443

Additional environment details (Operating System, Cloud provider, etc):

Ubuntu 18.04 Datadog-agent: 7.20.2 and 7.21.1

[vascop]

I've hit the same issue when switching to the EU datacenter. After configuring logs_dd_url I had to manually set use_http.

[Izual750]

I encounter the same issue on my side, migrating from the US to EU Endpoint, I had to set use_http: true on the configuration, and configure logs_dd_url explicitly

Thanks @vascop :)

[tlunsfordCXP]

We were having similar issues in kubernetes and found that configuring for UDS solved it. Datadog has documentation on how to configure for UDS - pretty straightforward.