[Feature] Add textfile collector support

[RaymondMouthaan]

Hello Lucas,

Via @carlosedp I came across rpi-exporter to get the temperature of raspberry pi's. Exactly what I needed! Thanks for that!

Currently I am using carlosedp/docker-arm_exporter to deploy a container on each of my docker nodes to get cpu temperature from each (raspberry pi) node. This works as expected, but it reports the ip address of the container instead of the node where it is running.

rpi_cpu_temperature_celsius{instance="10.0.1.26:9243",job="arm-exporter"} 55.843 rpi_cpu_temperature_celsius{instance="10.0.1.27:9243",job="arm-exporter"} 48.312 rpi_cpu_temperature_celsius{instance="10.0.1.28:9243",job="arm-exporter"} 56.382 rpi_cpu_temperature_celsius{instance="10.0.1.29:9243",job="arm-exporter"} 56.92 rpi_cpu_temperature_celsius{instance="10.0.1.30:9243",job="arm-exporter"} 57.996

Would it be possible to add the textfile-collector feature (or similar) like node-exporter to rpi_exporter, so that we can pass the hostname and node_id to rpi_exporter?

The way it works for node_exporter is:

docker compose file:

node-exporter:
    image: raymondmm/node-exporter
    environment:
      # pass the nodeid as a environment var
      - NODE_ID={{.Node.ID}}
    volumes:
      # pass the hostname of the host to container
      - /etc/hostname:/etc/nodename:ro
    command:
      # tell node_exporter where prom files can be found 
      - '--collector.textfile.directory=/etc/node-exporter/'
    ports:
      - 9100:9100
    networks:
      - monitoring-net
    deploy:
      mode: global
      # placement:
      #  constraints:
      #    # - node.role == manager
      #    - node.hostname == docker1
      restart_policy:
        condition: on-failure

docker-entrypoint.sh

#!/bin/sh -e

NODE_NAME=$(cat /etc/nodename)
echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom

set -- /bin/node_exporter "$@"

exec "$@"

The above script creates a file node-meta.prom:

cat /etc/node-exporter/node-meta.prom
node_meta{node_id="c6qrb488g71accirnhe1n7qqb", container_label_com_docker_swarm_node_id="c6qrb488g71accirnhe1n7qqb", node_name="docker1"} 1

This file gets read by node_exporter and makes it possible to query with Prometheus like this

sum(node_memory_MemAvailable * on(instance) group_left(node_id, node_name) node_meta) by (node_id, node_name)
{node_id="wrdvtftteo0uaekmdq4dxrn14",node_name="docker1"} 889450496
{node_id="moggm3uaq8tax9ptr1if89pi7",node_name="docker2"} 1404162048
{node_id="vkdfx99mm5u4xl2drqhnwtnsv",node_name="docker3"} 1406574592

Instead of:

sum(node_memory_MemAvailable) by (instance)
{instance="10.0.0.5:9100"} 889450496
{instance="10.0.0.13:9100"} 1404162048
{instance="10.0.0.15:9100"} 1406574592

Thanks in advanced, Raymond

[carlosedp]

Just remember to keep instance tag since it is important for deployments in Kubernetes where we can aggregate info from multiple targets by instance.

[RaymondMouthaan]

@carlosedp from the above node_exporter solution the instance tag is still available so it would be a problem when using Kubernetes.

[lukasmalkmus]

Hi Raymond, implementing something like this doesn't sound too hard to be honest. But would it be sufficient to just add something like the version collector most exporters use instead of a full-fledged file collector like node_exporter has it?

For example version collector exports the exporter_build_info metric for many exporters. For this exporter the metric name is rpi_exporter_build_info{...} and for e.g. the node_exporter it could be node_exporter_build_info{branch="HEAD",goversion="go1.9.1",instance="demo.robustperception.io:9100",job="node",revision="6e2053c557f96efb63aef3691f15335a70baaffd",version="0.15.0"}.

We could introduce a new metric named something like rpi_exporter_system_info which hashostname and node_id as labels.

On the other hand, just reading the metric (you already generate) from the file might be easy too 🤔 I'll take a look at the textfile collector implementation in node_exporter and bring up a proposal in the coming days.

[lukasmalkmus]

Oh and FYI: I made up this gist for people who are already running node_exporter and don't want to run another exporter just for monitoring the CPU temperature. It's a simple one-liner in your crontab which writes the temperature to a file which can then be collected by node_exporters file collector.

[RaymondMouthaan]

Lucas, thanks for the quick response! I guess the textfile collector could be an overkill, but i needed some kind of reference to explain what i wanted. If you have a solution / purposal, I am all ears :)

[lukasmalkmus]

Yes but thinking about it, the textfile collector example isn't that bad at all. This would allow anybody to add any kind of custom metric to the rpi_exporter.

[RaymondMouthaan]

I just had a quick look at your gist, but the node_exporter image i am running doesn't have vcgencmd installed. For cause I could add it to my image but I think Ill wait for the rpi-exporter solution :-)

[RaymondMouthaan]

Hai @lukasmalkmus have you made any progress on this? :-)

[lukasmalkmus]

Sorry for the late response, I have a tight schedule at the moment. So I took a look at the textfile collector implementation of node_exporter and I think it would be easiest to just copy it into this project. Since the node_exporter is Apache 2.0 licensed, this shouldn't be a problem when adding the appropriate notice to our modification.

Anything to add @RaymondMouthaan ?

[RaymondMouthaan]

Lucas, I think it would be sufficient when you add the textfile collector, this would make it possible to parse the node name like described in the first post.

When your implementation is ready, Carlos or myself can improve and test the docker image.

Let me know when you have your release ready.

Thanks again!

[lukasmalkmus]

Hi @RaymondMouthaan , I prepared #3. Besides adding the textfile collector, I also refactored some stuff. Can you checkout the develop branch, build from source and try for yourself?

[RaymondMouthaan]

Hi @lukasmalkmus, I like to inform you that your effort regarding the textfile collector paid off! I've build and tested it with my docker swarm.

This grafana graph shows now per node it's temperature based upon it's host name instead of ip address.

Can you please release the latest version of rpi_exporter? Thanks for this improvement! 🥇 👍

@carlosedp, I've created an fork of your project and modified it where needed to support this feature. After the release by @lukasmalkmus, I'll do some small changes and a PR to yours for merging.

[lukasmalkmus]

@RaymondMouthaan @carlosedp I merged the develop branch and tagged the latest release. Binaries are uploaded. Feel free to contact me if anything is missing.

[RaymondMouthaan]

@lukasmalkmus & @carlosedp, i like to inform you that it works as expected with the new docker image by Carlos. I've removed mine to avoid duplicates.

Thanks both for implementing this feature!