search

kumahq / kuma

🐻 The multi-zone service mesh for containers, Kubernetes and VMs. Built with Envoy. CNCF Sandbox Project.
https://kuma.io/install
Apache License 2.0
3.64k stars 331 forks source link

503 Errors from Datadog Remote Configuration #11632

Open healthy-pod opened 2 weeks ago

healthy-pod commented 2 weeks ago

What happened?

[reporting for future searches] Started getting the following error after upgrading to v2.7.7 even though we don't use DD remote config:

[source/extensions/tracers/datadog/logger.cc:23] Unexpected Remote Configuration status 503 with body (if any, starts on next line):
upstream connect error or disconnect/reset before headers. reset reason: connection termination

This can be fixed by setting DD_REMOTE_CONFIGURATION_ENABLED to false. This can be set in KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_ENV_VARS environment variable in the zone control plane. Helm chart example:

envVars:
    KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_ENV_VARS: "DD_REMOTE_CONFIGURATION_ENABLED:false"

Note that this started happening due to a change in envoy not kuma and the same error appeared in other meshes using envoy as a proxy.

slonka commented 1 week ago

triage: was this erroring by default? or did you turn something on? if by default we'll disable it