How to run in docker?

[omegazeng]

I run KA on CentOS on the bare metal.

I want to run keepalived_exporter in docker like node_exporter.

Does it support?

---

This is my attempt.

cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 

keepalived -v
Keepalived v1.3.5 (03/19,2017), git commit v1.3.5-6-g6fa32f2

Copyright(C) 2001-2017 Alexandre Cassen, <acassen@gmail.com>

Build options:  PIPE2 LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_PREF FRA_OIFNAME FRA_SUPPRESS_PREFIXLEN FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK LIBIPTC LIBIPSET_DYNAMIC LVS LIBIPVS_NETLINK VRRP VRRP_AUTH VRRP_VMAC SOCK_NONBLOCK SOCK_CLOEXEC FIB_ROUTING INET6_ADDR_GEN_MODE SNMP_V3_FOR_V2 SNMP SNMP_KEEPALIVED SNMP_CHECKER SNMP_RFC SNMP_RFCV2 SNMP_RFCV3 SO_MARK

docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2f86
 Built:             Tue Oct  8 00:58:10 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2f86
  Built:            Tue Oct  8 00:56:46 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker run --rm -d -v /lib/modules:/lib/modules:ro,rslave -v /proc:/proc --pid=host --net=host keepalived_exporter:0.1.0
b180a7679730d67410a8c5e0aaa422f98a3f8b6647c27b27aaf845a33eb6ba37

docker logs b180a7679730d67410a8c5e0aaa422f98a3f8b6647c27b27aaf845a33eb6ba37
2019/12/05 07:38:38 Providing metrics at :9650/metrics
2019/12/05 07:38:51 keepalived_exporter: operation not permitted
2019/12/05 07:39:20 keepalived_exporter: operation not permitted
2019/12/05 07:39:50 keepalived_exporter: operation not permitted

curl localhost:9650/metrics
# HELP keepalived_up Status
# TYPE keepalived_up gauge
keepalived_up 0

[gen2brain]

Exporter works by sending SIGNAL to keepalived process, and then decoding/parsing text or json files in /tmp directory that KA will write after it receives signal.

In this case, you don't have permission to send SIGNAL to process, CAP_KILL capability can allow you to do that. But then, you will have problems reading /tmp/keepalived.data file, not sure if there is any capability to work around that.

[gen2brain]

Ok, there is CAP_DAC_READ_SEARCH, so try to add this to docker command you are running: --cap-add DAC_READ_SEARCH --cap-add KILL

[omegazeng]

Works using --cap-add NET_ADMIN -u 0 -u 0 is needed, must be root.

docker run --rm -d -v /lib/modules:/lib/modules:ro,rslave -v /tmp:/tmp:ro,rslave -p 9650:9650 --pid=host --cap-add NET_ADMIN -u 0 keepalived_exporter:0.1.0
98f037dd7aaa487f1b2108f1dcaee1996808ac8dccbea691839c21f49357ad93

curl localhost:9650/metrics
# HELP keepalived_up Status
# TYPE keepalived_up gauge
keepalived_up 1
# HELP keepalived_vrrp_addr_list_err Address list errors
# TYPE keepalived_vrrp_addr_list_err counter
keepalived_vrrp_addr_list_err{intf="",name="VI_1",state="INIT",vrid="0"} 0
...

---

I tried using --cap-add DAC_READ_SEARCH --cap-add KILL -u 0, not working. The same error message: "keepalived_exporter: operation not permitted"

--cap-add KILL is added by default. https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

Maybe --cap-add NET_ADMIN does not recommend, but which one is needed?

[gen2brain]

It is probably not the same error you are getting, the first one was from SIGNAL, and with -u 0 it is probably some IPVS/LVS access, where it worked with NET_ADMIN. I will add descriptions for errors, so it will be easier to identify the problem in the future.

Btw. not sure why you insist on docker if you already use bare metal. It is not really recommended for node_exporter, and both node and keepalived exporters are single static binaries without dependencies, just write simple systemd services and run it like that. Oh, and you can run keepalived as a user and not root, I use it like that, and keepalived_exporter runs as the same user.

[omegazeng]

Ok, Before I use Helm to deploy Prometheus in Kubernetes, node_exporter is running in docker.

Thank you for your suggestion! I will change to use Systemd.

[franky8109]

It is probably not the same error you are getting, the first one was from SIGNAL, and with -u 0 it is probably some IPVS/LVS access, where it worked with NET_ADMIN. I will add descriptions for errors, so it will be easier to identify the problem in the future.

Btw. not sure why you insist on docker if you already use bare metal. It is not really recommended for node_exporter, and both node and keepalived exporters are single static binaries without dependencies, just write simple systemd services and run it like that. Oh, and you can run keepalived as a user and not root, I use it like that, and keepalived_exporter runs as the same user.

Hello gen2brain, I have similar challenge as this post so i am trying to perform your suggestion to run keepalived with same user as exporeter but can not find the way. Could you let me (us) know how exactly you did that as when i change the user and give write permission i receive an output stating Keepalived[20693]: pidfile_write : Cannot open /var/run/keepalived.pid pidfile.

Thank you

[gen2brain]

Description=LVS and VRRP High Availability Monitor
After=syslog.target network-online.target

[Service]
Type=forking
User=someuser
Group=someuser
PermissionsStartOnly=true
PIDFile=/var/run/keepalived/keepalived.pid
KillMode=process
EnvironmentFile=-/etc/sysconfig/keepalived
ExecStartPre=/sbin/setcap cap_setgid,cap_sys_nice,cap_net_raw,cap_net_admin=eip /usr/sbin/keepalived
ExecStart=/usr/sbin/keepalived -f /etc/keepalived/keepalived.conf -p /var/run/keepalived/keepalived.pid -r /var/run/keepalived/vrrp.pid -c /var/run/keepalived/checkers.pid -S 3 $KEEPALIVED_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

And make sure to load kernel modules on startup, KA will not be able to load modules it needs if they are not already loaded.

Edit: You will probably also want to configure tmpfiles.d, something like this: D /var/run/keepalived 0710 someuser someuser -

[franky8109]

Appreciate!! i will test and revert if needed.

BR,