Does not work so great inside a docker container

[camerondavison]

Currently it reports /etc/hostname and /etc/hosts and /etc/resolv.conf as mount points inside the docker container, and then reports / as just the filesystem that the docker container is mounted on.

It would be nice to be able to report the mounts from the host machine, or at least have some configuration option where you can do that similar to cAdvisor.

[matthiasr]

Does it work to bind-mount /proc and /sys into the container?

[camerondavison]

docker run --rm -ti -p 9100:9100 prom/node-exporter

$ curl -s localhost:9100/metrics | egrep '^node_filesystem_free'
node_filesystem_free{filesystem="/"} 1.0477051904e+10
node_filesystem_free{filesystem="/etc/hostname"} 1.0477051904e+10
node_filesystem_free{filesystem="/etc/hosts"} 1.0477051904e+10
node_filesystem_free{filesystem="/etc/resolv.conf"} 1.0477051904e+10

docker run --rm -ti -p 9100:9100 --volume=/sys:/sys:ro --volume=/proc:/proc:ro prom/node-exporter

same

docker run --rm -ti -p 9100:9100 --volume=/sys:/sys:ro --volume=/proc:/proc:ro --volume=/:/rootfs:ro prom/node-exporter

sort of works but has a bunch of junk like below that I would probably not like to have

$ curl -s localhost:9100/metrics | egrep '^node_filesystem_free'
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/12750f959cc8ead2298a7f8684a6e8d2b2ac1fb88853945a055debeb150d5484/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/1cf04940027f49751de0b4152cb5ddcae75a9377ea1ba3e27b081110a43afb99/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/25fe6dd3e7aa9344c5150dabc931da7090c91d063780e7b0806e013c22fab847/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/2ca45cb64bd8a03f508f01f3a328336a8403948190a5966864f00375c2061128/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/517e01efdc1d013d45085fac163fe65415a3d3a5e6037442a2de1b1aeebfbb3b/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/6a23736beed6e76bed99f4d73bfecf007c62427a1d882ee52cdccb9fadc1568f/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/72047aa2084de6e64d904b4ec223d5e11586290fb6a2a32f02f6a079d65bf248/merged"} 1.047703552e+10
node_filesystem_free{filesystem="/rootfs/var/lib/docker/overlay/84827b2dba0e0383d767f6a79db7086a219b1736bd3494273d9ac8b120b9d7b3/merged"} 1.047703552e+10

[matthiasr]

I'm afraid that is just the reality of Docker (lots of mounts).

In general, trying to get information about the actual machine from inside a container will be quite awkward regardless; I would treat it as part of the platform and run it bare. At most you could limit its CPU and memory usage, but if you namespace it off it just won't be able to tell you much.

[camerondavison]

cAdvisor does not have this problem :/

[juliusv]

But, but... cAdvisor only exports cgroup container stats, not fileystem stats, right? The cAdvisor Docker launch instructions say:

sudo docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:rw \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  google/cadvisor:latest

The --volume=/sys:/sys:ro flag mounts the host's sysfs (containing cgroup information under /sys/fs/cgroup) into the cAdvisor container, making all the cgroup information accessible to cAdvisor. Perhaps a similar thing can be done by mounting all the filesystems you are interested in into the container, but it's not going to be as easy as with the cgroup information, as that only needs to mount in one FS with all the cgroup info.

[juliusv]

Ah ok, it also does FS stats, but only for filesystems mounted into each container?

[juliusv]

Or maybe the --volume=/:/rootfs:ro also allows it to export host filesystem stats, not sure.

[matthiasr]

(I saw this in email but the comment is gone?)

Mounting the host's /proc into a different path in the container would definitely be a good way.

Currently, many of the collectors hardcode /proc, they should instead read this from a (one global) flag. This would also make e.g. testing much more consistent.

[grobie]

All the procfs code should be moved to our procfs lib anyway, which supports changing the mounting point already.

[discordianfish]

Agreed, we should switch to procfs lib and make it possible to specify a alternative root directory (--fs.root /rootfs or similar) and only expose metrics for filesystems mounted in that directory. To make things nice, we can strip the rootfs path from the filesystem path.

[juliusv]

Volunteers? :)

[juliusv]

We have flags now for setting the procfs and sysfs locations, so as long as you get those two mounted into your Docker container somehow, you should be able to run it on Docker: https://github.com/prometheus/node_exporter/blob/master/collector/paths.go

[superdump]

Do you necessarily need to mount to some path in the container that is not /sys and /proc?

[juliusv]

No, those two should be sufficient.

[superdump]

I meant: is there a problem with -v /sys:/sys -v /proc:/proc? Or does one have to mount to different paths in the container? Is :ro needed for any security reasons?

[superdump]

-v /proc:/proc is disliked:

Cannot start container b2cdc636e8be690293659044cfae954e198a581d65b9e9131897c7be716504bd: [8] System error: "/var/lib/docker/aufs/mnt/b2cdc636e8be690293659044cfae954e198a581d65b9e9131897c7be716504bd/proc" cannot be mounted because it is located inside "/proc"

[juliusv]

I'm afraid I won't be able to help here since I don't do much with Docker. As long as you can somehow mount both FSes into your container and point the node exporter flags there, you should be fine though (i.e. I don't think mounting ro is strictly needed, though it might not hurt).

[discordianfish]

@superdump Yes, I guess mounting to /proc doesn't work. Just use some other directory. That being said, we should consider to support some filtering/rewriting of the mountpoints for this case. Basically just strip the path to the container root.

[mbrooks]

The below got me the info I needed for the most part. I built my dashboard showing disk space available by using 'node_filesystem_avail{mountpoint="/rootfs"}' as the query. It seems to work pretty well.

docker run -d -p 9100:9100 -v "/proc:/host/proc" -v "/sys:/host/sys" -v "/:/rootfs" --net="host" prom/node-exporter -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

It's kind of annoying that root point shows up as /rootfs and that's definitely not standard, but it works as far as I can tell for CPU and Disk stats!

My only suggestion is that node_explorer may need to rethink it's docker strategy a bit to maybe include some of the above commands by default and require certain volumes are mounted. Info isn't going to be in the standard places and parsing may need to be different as well for docker images to keep things consistent in a docker and non-docker world.

[discordianfish]

@mbrooks PRs welcome :) I agree that it makes sense to:

• add a option to strip a prefix from mountpoints
• require and check that things are bind-mounted
• update the README to explain all this a bit more

[flypenguin]

I really think this is highly inconvenient. Having a "disk free" item should be the most basic form of monitoring, and node-exporter fails in doing so. IMHO is a major issue because it kinda renders disk stats unusable (I really don't want to find out what filesystem "/etc/resolv.conf" actually is, and everyone looking at any graphs is bound to wonder)

I am trying now to run this as standalone go application to fix it, luckily it's Go so it doesnt have dependencies. Let's see how that goes.

[brian-brazil]

The node exporter does provide disk free stats, your problem is that you're trying to run it inside a system designed to prevent the node exporter getting to things like disk stats - and which also uses a fair amount of bind mounts (which is what /etc/resov.conf is here, we should see if we can exclude bind mounts as they don't make sense to export).

[discordianfish]

Agreeing with @brian-brazil here. In general I would advise against running the node_exporter as Docker container. In my infrastructures I usually bake the node-exporter in the host/OS image. That being said, I think it's reasonable to support some 'rewriting' and filtering of mountpoints given how many people want to run it in Docker. So @flypenguin feel free to submit a PR adding such functionality.

PS: If you need this but can't fix it on your own, I'm sure we can find someone you can pay to do so ;)

[flypenguin]

then my pull request would be to remove that paragraph "you can use docker to get started" replacing it with "you can use docker, but it is discouraged because of weird metric labels". ;)

2016-03-16 11:12 GMT+01:00 discordianfish notifications@github.com:

Agreeing with @brian-brazil https://github.com/brian-brazil here. In general I would advise against running the node_exporter as Docker container. In my infrastructures I usually bake the node-exporter in the host/OS image. That being said, I think it's reasonable to support some 'rewriting' and filtering of mountpoints given how many people want to run it in Docker. So @flypenguin https://github.com/flypenguin feel free to submit a PR adding such functionality.

PS: If you need this but can't fix it on your own, I'm sure we can find someone you can pay to do so ;)

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/prometheus/node_exporter/issues/66#issuecomment-197245699

[superdump]

I run it in a docker container without any significant problems. If you have infrastructure for managing docker containers and you try to deploy everything using docker containers then it can make sense to run it there anyway for consistency with the rest of your system.

[juliusv]

Btw., https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-using-docker-on-ubuntu-14-04 also has some instructions on how to set up the Node Exporter in Docker. It's the closest I could make it compared to running it directly on the host system (which is still recommended). Basically:

docker run -d -p 9100:9100 -v "/proc:/host/proc" -v "/sys:/host/sys" -v "/:/rootfs" --net="host" prom/node-exporter -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

[flypenguin]

I'll just run it natively, I'm still using "normal" Linux installations (not something super-stripped-down where I potentially can't do this ;). Still, thanks for your effort, I'll bookmark this in case I'll hit this again somewhere else :)

2016-03-16 13:39 GMT+01:00 Julius Volz notifications@github.com:

Btw., https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-using-docker-on-ubuntu-14-04 also has some instructions on how to set up the Node Exporter in Docker. It's the closest I could make it compared to running it directly on the host system (which is still recommended). Basically:

docker run -d -p 9100:9100 -v "/proc:/host/proc" -v "/sys:/host/sys" -v "/:/rootfs" --net="host" prom/node-exporter -collector.procfs /host/proc -collector.sysfs /host/proc -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/prometheus/node_exporter/issues/66#issuecomment-197300231

[mfournier]

So it looks like the hosts' /proc inside a container doesn't contain the same data depending on how you specify the volumes mapping.

I didn't compare each file, but /proc/mounts, which is used to list the filesystems definetely differs:

# grep ext /proc/mounts 
/dev/vda1 / ext3 rw,relatime,errors=remount-ro,data=ordered 0 0
/dev/vdb /var/lib/docker ext4 rw,noatime,commit=600,data=ordered 0 0

# docker run -it --rm -v /proc:/host/proc debian:jessie grep ext /host/proc/mounts
/dev/vdb /etc/resolv.conf ext4 rw,noatime,commit=600,data=ordered 0 0
/dev/vdb /etc/hostname ext4 rw,noatime,commit=600,data=ordered 0 0
/dev/vdb /etc/hosts ext4 rw,noatime,commit=600,data=ordered 0 0

# docker run -it --rm -v /:/host debian:jessie grep ext /host/proc/mounts
/dev/vda1 /host ext3 rw,relatime,errors=remount-ro,data=ordered 0 0
/dev/vdb /host/var/lib/docker ext4 rw,noatime,commit=600,data=ordered 0 0
/dev/vdb /host/var/lib/docker/aufs ext4 rw,noatime,commit=600,data=ordered 0 0
/dev/vdb /etc/resolv.conf ext4 rw,noatime,commit=600,data=ordered 0 0
/dev/vdb /etc/hostname ext4 rw,noatime,commit=600,data=ordered 0 0
/dev/vdb /etc/hosts ext4 rw,noatime,commit=600,data=ordered 0 0

So the former way of doing makes node_exporter skip filesystems not "mounted" inside the scope of the container.

[bonovoxly]

@juliusv it appears to be a nice fix. However, I'm seeing behavior that completely ignores the -collector.procfs /host/proc option. Either that, or I'm doing something wrong. Maybe someone can help me figure it out. Here's what I'm seeing...

I did a little clicking around and it appears that the procfs has a hardcoded default:

https://github.com/prometheus/node_exporter/blob/bce2e0c51393be4fa8e96b3c2843623135ee4d0f/vendor/github.com/prometheus/procfs/fs.go

const DefaultMountPoint = "/proc"

If we look in:

https://github.com/prometheus/node_exporter/blob/bce2e0c51393be4fa8e96b3c2843623135ee4d0f/vendor/github.com/prometheus/procfs/proc.go

Here's the code that references the hardcoded value:

// AllProcs returns a list of all currently avaible processes under /proc.
func AllProcs() (Procs, error) {
    fs, err := NewFS(DefaultMountPoint)
    if err != nil {
        return Procs{}, err
    }
    return fs.AllProcs()
}

This behavior seems to reflect what I've seen when running:

docker run -d -p 9100:9100 -v "/proc:/host/proc" -v "/sys:/host/sys" -v "/:/rootfs" --net="host" prom/node-exporter -collector.procfs="/host/proc" -collector.sysfs /host/sys -collector.filesystem.ignored-mount-points "^/(sys|proc|dev|host|etc)($|/)"

When running:

curl http://localhost:9100/metrics|grep node_procs

I see something like (shortened the output):

# TYPE node_procs_running gauge
node_procs_running 1

The docker container is running one process. The host is running a lot more. It appears that it is not using the host volume mount of /host/proc, but instead is usinbg a hard coded /proc. Wondering if anyone else has seen this behavior...

[cdrage]

@mfournier

Hey man, were you ever able to fix the issue with /proc mounting showing different values?

I've tried -v /proc/net/dev:/host/prov/net/dev:ro with Docker and for some odd reason it shows different values than what would of been shown on the host /proc/net/dev

[grobie]

@bonovoxly The procfs function you linked there is only a convenience function, there is also an additional one operation on a custom FS type.

Please also note that the stat collector reimplemented the proc parsing and doesn't use the procfs library.

[cdrage]

Blargh. Seems like a limitation of Linux namespaces for getting /proc information... https://github.com/docker/docker/issues/13398

[zerthimon]

+1

[discordianfish]

So not much we can do about this issue. I've updated the README to make it easier for people to figure out how to run it on Docker as well as state that it's not recommended: #376

Beside that, I'm still open to have some 'stripping' of a prefix but everyone who was interested in this happily migrated to running the exporter on the host, so I'd like to close this issue after merging #376.

If somebody is still interested in stripping the bind-mount prefix from the metrics, please open a new issue for that - or submit a PR.

[discordianfish]

As discussed, I'm going to close this issue now that #376 is closed. It documents the best way to run the exporter as Docker container. Further improvement like stripping some prefix from the labels is something, if needed, can be tracked in a new issue.

[discordianfish]

While we still recommend running the node-exporter on the host and not in a container runtime, many people (including myself) do so. So I think it's time we revisit this issue and use it to track other related things like:

• 502 Disks mounted after starting the container don't show up

[discordianfish]

I think there are other issues beside the names to address, like the issue linked above. I know Red Hat is working on some solutions to that which involve running the node-exporter in the host namespace, that might solve all these issues.

[discordianfish]

@s-urbaniak Still waiting for you guys take on it btw :)

[dash042]

still not working: node-exporter in Docker container (k8s) with variable node_filesystem_size_bytes will only show 10GB - so the docker mapped volume - not the total disk size of the NODE:

/dev/mapper/docker-253:0-201434461-e80c535ec0ec3840bef82cac01ae9d4ca47550036ee8a8cc222aa722aaf919a2
                         10.0G     55.5M      9.9G   1% /

Solution: use quay.io/prometheus/node-exporter:v0.15.0 and node_filesystem_size after mounting / to /rootfs... and filtering on this mountpoint

Definetly not perfect - but I have no idea why the newer versions outputs fewer data...

[discordianfish]

@dash042 That is already documented in the README

[dt-rush]

Agreeing with @brian-brazil here. In general I would advise against running the node_exporter as Docker container. In my infrastructures I usually bake the node-exporter in the host/OS image.

The issue is, docker is used precisely because it makes it possible to design and run your infrastructure in such a way that you don't have to have control of (and you may not have control of) of the base image to the point you can decide to "bake in" a binary like the node exporter.

Docker creates "containers" for process trees using cgroups and namespaces. It is not necessarily meant in all cases and above all to block access, if desired, to the host. It's to be expected that in the modern world, people will deploy this service in a containerized manner. I think the solution this thread hit upon is fine enough; Bind-mounting is a well-established part of docker practices.

[dt-rush]

There is only one thing I would add for future reference of anybody finding this thread, or perhaps this might even warrant inclusion in the README:

If the --pid="host" param is not provided, the metrics for avail_bytes will not be labelled properly. You will see:

node_filesystem_avail_bytes{device="/dev/nvme0n1p2",fstype="ext4",mountpoint="/etc/hostname"} 1.3968082944e+11
node_filesystem_avail_bytes{device="/dev/nvme0n1p2",fstype="ext4",mountpoint="/etc/hosts"} 1.3968082944e+11

instead of:

node_filesystem_avail_bytes{device="/dev/nvme0n1p2",fstype="ext4",mountpoint="/"} 1.3967706112e+11

[dt-rush]

... This is particularly troublesome if you want to run systemd in your container (via an entrypoint which runs /sbin/init) (see for example this base image that we use for our services), and want to have the node-exporter be run as a systemd unit inside that container, since --pid="host" will cause the container to fail to start with the logs showing:

Systemd init system enabled.
Couldn't find an alternative telinit implementation to spawn.

[discordianfish]

@dt-rush The readme also tells people to use pid=host. If you have a suggestion how to solve this without using host pid namespace, I'm all ears.

[n0nvme]

Currently docker swarm does not support using host pid(see https://github.com/docker/docker.github.io/issues/5624#issuecomment-354102472). Are there any workarounds to monitor nodes with node-exporter started as swarm service in global mode?

[SuperQ]

Use Ansible to deploy the node_exporter instead of using Docker.

[SuperQ]

I think we've documented enough of the Docker issues in the README at this point. For normal Docker use, the node_exporter works well enough. Additional issues need to be filed with Docker, as they are not node_exporter bugs.