search

google / cadvisor

Analyzes resource usage and performance characteristics of running containers.
Other
17.14k stars 2.32k forks source link

Unable to get network stats from pid #1522

Open gl328518397 opened 8 years ago

gl328518397 commented 8 years ago

my OS is redhat7, docker is 1.8 my cmd is " docker run --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --volume=/cgroup:/cgroup:ro \ --publish=8096:8080 \ --detach=true \ --link influxsrv:influxsrv \ --privileged=true \ google/cadvisor:latest \ -v=2 \ -storage_driver=influxdb \ -storage_driver_db=cadvisor \ -storage_driver_host=127.0.0.1:8086 "

i say logs -->> Unable to get network stats from pid 34215: couldn't read network stats: failure opening /rootfs/proc/34215/net/dev: open /rootfs/proc/34215/net/dev: no such file or directory I1101 06:01:03.346357 1 helpers.go:101] Unable to get network stats from pid 32943: couldn't read network stats: failure opening /rootfs/proc/32943/net/dev: open /rootfs/proc/32943/net/dev: no such file or directory I1101 06:01:04.145622 1 helpers.go:101] Unable to get network stats from pid 34215: couldn't read network stats: failure opening /rootfs/proc/34215/net/dev: open /rootfs/proc/34215/net/dev: no such file or directory I1101 06:01:05.720426 1 helpers.go:101] Unable to get network stats from pid 32943: couldn't read network stats: failure opening /rootfs/proc/32943/net/dev: open /rootfs/proc/32943/net/dev: no such file or directory I1101 06:01:07.009749 1 helpers.go:101] Unable to get network stats from pid 32943: couldn't read network stats: failure opening /rootfs/proc/32943/net/dev: open /rootfs/proc/32943/net/dev: no such file or directory I1101 06:01:07.710892 1 helpers.go:101] Unable to get network stats from pid 34215: couldn't read network stats: failure opening /rootfs/proc/34215/net/dev: open /rootfs/proc/34215/net/dev: no such file or directory I1101 06:01:09.289906 1 helpers.go:101] Unable to get network stats from pid 34215: couldn't read network stats: failure opening /rootfs/proc/34215/net/dev: open /rootfs/proc/34215/net/dev: no such file or directory I1101 06:01:10.660529 1 helpers.go:101] Unable to get network stats from pid 32943: couldn't read network stats: failure opening /rootfs/proc/32943/net/dev: open /rootfs/proc/32943/net/dev: no such file or directory

dashpole commented 7 years ago

what cadvisor version are you running?

Dennisparchkov commented 5 years ago

I'm getting the same issue as described:

I am am running latest cadvisor in a container in Centos7, Docker version 18.03.1.

When i restart one of the containers i am monitoring using docker restart ... i get the following in the cadvisor logs:

 Unable to get network stats from pid 18927: couldn't read network stats: failure opening /│            - /var/lib/docker/:/var/lib/docker:ro
rootfs/proc/18927/net/dev: open /rootfs/proc/18927/net/dev: no such file or directory

Result is no Network stats but has_network is still true. All other stats still load (CPU, memory), only issue is Network

I assume that docker restart changes the pid of the container.

dashpole commented 5 years ago

Is the issue temporary or permanent?

Dennisparchkov commented 5 years ago

permanent issue, every container (that i want to monitor) has this issue when it restarts.

dashpole commented 5 years ago

Yeah, this looks like a bug with the interaction of cAdvisor with docker restart. cAdvisor assumes the container process stays the same for the lifetime of the container. I ran a quick test, and docker restart keeps the container id the same, but changes the PID of the container.

There are a few things we could do:

Does the PID change only when docker restart is used, or can it change for other reasons (e.g. docker container exits, but a restart policy was specified)?

Dennisparchkov commented 5 years ago

Actions like starting and stopping containers docker stop and docker start also change its PID.

I was thinking of using the event mechanism from the go docker library used (https://godoc.org/github.com/docker/docker/client#Client.Events) for monitoring these types of events that will change the PID and act accordingly.

For this specific case when a restart event is triggered the PID of the container should be updated.

dashpole commented 5 years ago

That might be a better solution than a periodic refresh. I would be curious to see how complex that is to implement.