WARN logging occurs on containers without IPs, flooding logs every minute

timvanderkooi commented 5 years ago

Describe what happened:

Every minute, the agent will log something like:

2019-06-28 20:57:02 UTC | PROCESS | WARN | (pkg/util/docker/containers.go:223 in parseContainerNetworkAddresses) | Unable to parse IP: for container: /twistlock_defender_18_11_128"

This started occurring after the latest update to 6.12. Since we have our agent set to WARN logging, this floods our logging every minute with these types of logs from all of containers without IPs.

Describe what you expected:

This logging should be INFO instead of WARN. No IPs is an expected part of our deployment since some containers do not require an IP.

Steps to reproduce the issue:

Create a container without an IP

Additional environment details (Operating System, Cloud provider, etc):

DylanLovesCoffee commented 5 years ago

hey @timvanderkooi, thanks for raising the issue. We're noticing that the logs are being populated for pause containers, since their network mode is none when coupled with K8s (since networking is delegated to an underlying network plugin). This was introduced in 6.12.0, and although the logs are harmless we'll work on this and update you here when a change is made.

xlucas commented 5 years ago

A fix has been merged for this, it will be available in the next release. Therefore, I think we can close this issue now.

timvanderkooi commented 5 years ago

@xlucas thanks! Appreciate it.

davidmoshal commented 4 years ago

Odd, Datadog itself doesn't log errors, but when the same server is also instrumented, eg with LogDNA, then LogDNA logs these errors from the datadog-agent:

Dec 10 22:15:01 1c714f9dd34a datadog-agent INFO UTC | PROCESS | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /frontend will be missing network info: %!s(<nil>)
Dec 10 22:15:01 1c714f9dd34a datadog-agent INFO UTC | PROCESS | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /engine will be missing network info: %!s(<nil>)
Dec 10 22:15:01 1c714f9dd34a datadog-agent INFO UTC | PROCESS | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /nginx will be missing network info: %!s(<nil>)
Dec 10 22:15:01 1c714f9dd34a datadog-agent INFO UTC | PROCESS | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /redis will be missing network info: %!s(<nil>)
Dec 10 22:15:08 1c714f9dd34a datadog-agent INFO UTC | CORE | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /frontend will be missing network info: %!s(<nil>)
Dec 10 22:15:08 1c714f9dd34a datadog-agent INFO UTC | CORE | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /engine will be missing network info: %!s(<nil>)
Dec 10 22:15:08 1c714f9dd34a datadog-agent INFO UTC | CORE | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /nginx will be missing network info: %!s(<nil>)
Dec 10 22:15:08 1c714f9dd34a datadog-agent INFO UTC | CORE | ERROR | (pkg/util/docker/containers.go:110 in ListContainers) | Failed to get host IPs. Container /redis will be missing network info: %!s(<nil>)

dockerpoa commented 4 years ago

I have the same error:

I'm using k8s and deployment datadog-agent:7.16.1

ghost commented 4 years ago

I have the same error and did some tests. I noticed that the problem occurs with Datadog 7 (installed using helm chart version 1.39.5). When I switched to Datadog 6.14.0, it was working fine.

rpf3 commented 4 years ago

I was getting this error when running agent version 6.12.0-jmx but fixed it by upgrading to 7.19.2-jmx.

DataDog / datadog-agent

WARN logging occurs on containers without IPs, flooding logs every minute #3796