oofnikj / docker-openwrt

OpenWrt running in Docker
GNU General Public License v2.0
317 stars 67 forks source link

Install monitoring with physical OpenWrt device #22

Closed darrepac closed 3 years ago

darrepac commented 3 years ago

Hello

I am a bit lost. I have already a running openwrt router so I am not interested by the openwrt part of this repo but the grafana part... yet it is in the same repo so I wonder if I can only use grafana part and how to install it. I guess that influxdb and grafana are part of the container so what IP adress should I put in the .env file... Well as you understand, I don't understand :)

oofnikj commented 3 years ago

Yes, it's possible.

The instructions for setting up the monitoring components are independent from the Docker stuff in the main part of the repository. You just need to make sure that your instances of InfluxDB and Grafana are properly accessible on the network.

If you use the Docker Compose script provided, you'll need to modify it slightly by removing the external: true specification on the openwrt-lan network which will instruct Docker to create a bridge network with this name instead of expecting it to exist already. In addition, you will need port forwarding so that the host on which the services are running can communicate properly with OpenWrt. Grafana by default listens on port 3000. InfluxDB listens by default on ports 8086 (API) and for collectd metrics on port 25826 per the config file influxdb.conf.

Let's say you are running OpenWrt on 192.168.0.1, and your monitoring server on 192.168.0.2. Your Docker Compose script might look something like this:

version: "3"
services:
  grafana:
    container_name: grafana
    image: grafana/grafana:7.3.5
    networks:
      openwrt-lan:
    ports:
      - "3000:3000"
    env_file:
      - .env
    volumes:
      - "./grafana/provisioning:/etc/grafana/provisioning"
      - "./grafana/dashboard-openwrt.json:/var/lib/grafana/dashboards/dashboard-openwrt.json"
    restart: always

  influxdb:
    container_name: influxdb
    image: influxdb:1.7
    networks:
      openwrt-lan:
    ports:
      - "8086:8086"
      - "25826:25826/udp"
    env_file: 
      - .env
    volumes:
      - "./influxdb/influxdb.conf:/etc/influxdb/influxdb.conf"
      - "./influxdb/types.db:/usr/share/collectd/types.db"
      - "influxdb:/var/lib/influxdb"
    restart: always

networks:
  openwrt-lan:

volumes:
  influxdb:

We omit the ipv4_address static configuration here and instead allow Docker to assign an IP address automatically to our containers. We don't really care what they are since we are forwarding ports to the host. You then point your OpenWrt router to send collectd metrics to 192.168.0.2:25826. With any luck, Grafana will show up at http://192.168.0.2:3000 and ask for the credentials you set in .env.

EDIT: add /udp for InfluxDB collectd port

darrepac commented 3 years ago

Am I right saying that in openwrt /etc/collectd.conf, I should add (per your example):

<Plugin network>
  Server "192.168.0.1.2" "25826"
</Plugin>

Because in iptmon, the doc is vague:

Furthermore, collectd can push data to InfluxDB

oofnikj commented 3 years ago

collectd is configured entirely through UCI. You shouldn't manually create a collectd.conf file on your OpenWrt device as UCI will generate this configuration on its own once the network export settings are configured.

In the docs "setup network export" section, the value that goes in $INFLUXDB_ADDRESS is the host name or IP address of the host running the InfluxDB container.

darrepac commented 3 years ago

Ok Grafana is up and running but the dashboard is void ("no data"). So I guess data is not coming into the database... On Openwrt side, it seems ok (192.168.0.13 is the pi where is docker grafana/influxdb running): image How can I check?

oofnikj commented 3 years ago

The "server host" should be the IP of the Pi running InfluxDB. If that's 192.168.0.13 then that's what you should fill in there.

Perhaps there's some firewall rule preventing communication between your OpenWrt router and the Pi?

If you run tcpdump on the Pi (replace $INTERFACE with your actual network interface name) you should see a burst of UDP data every couple of seconds on port 25826. On my setup it looks like this:

$ sudo tcpdump -i $INTERFACE udp port 25826
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br-fc2a68b4e3a6, link-type EN10MB (Ethernet), capture size 262144 bytes
08:31:17.578585 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1289
08:31:17.579144 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1312
08:31:17.579591 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1332
08:31:17.580020 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1336
08:31:17.580430 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1323
08:31:27.578135 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1322
08:31:27.578619 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1305
08:31:27.579112 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1303
08:31:27.579761 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1327
08:31:27.580323 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1323
08:31:27.581337 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1322
08:31:27.582480 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1307
08:31:37.578461 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1310
08:31:37.579082 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1336
08:31:37.579515 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1323
08:31:37.579929 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1322
08:31:37.580328 IP openwrt.home.41513 > 192.168.16.4.25826: UDP, length 1312

You can also confirm that the collectd database was created in InfluxDB properly by:

$ docker exec -it influxdb influx -database collectd
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> show measurements
name: measurements
name
----
conntrack_value
cpu_value
dns_queries
dns_responses
dns_value
...

Finally you can check your OpenWrt logs to see if there are any error messages from collectd with logread -e collectd.

darrepac commented 3 years ago

Ok thanks. On pi / tcpdump side, it sounds good:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:06:20.412834 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1342
09:06:20.423946 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1328
09:06:20.427106 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1311
09:06:20.430484 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1328
09:06:20.443974 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1333
09:06:20.447307 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1338
09:06:20.455858 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1305
09:06:30.416793 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1328
09:06:30.425755 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1342
09:06:30.429245 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1333
09:06:30.432699 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1304
09:06:30.436183 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1323
09:06:30.439398 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1325
09:06:30.442881 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1321
09:06:30.446665 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1339
09:06:30.450138 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1308
09:06:30.463649 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1310
09:06:30.466593 IP OpenWrt.lan.53354 > rpi-influxDB.lan.25826: UDP, length 1315

About collectd database in influxdb, this sounds less good:

pi@raspberrypi:~ $ sudo docker exec -it influxdb influx -database collectd
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> show measurements
>

And on openwrt side:

root@OpenWrt:~# logread -e collectd
root@OpenWrt:~#

I am really quite noob at docker, but are the ports correct?

pi@raspberrypi:~ $ sudo docker ps
CONTAINER ID   IMAGE                   COMMAND                  CREATED        STATUS        PORTS                                              NAMES
38f7675c5b27   influxdb:1.7            "/entrypoint.sh infl…"   12 hours ago   Up 12 hours   0.0.0.0:8086->8086/tcp, 0.0.0.0:25826->25826/tcp   influxdb
9a387d3a4703   grafana/grafana:7.3.5   "/run.sh"                12 hours ago   Up 12 hours   0.0.0.0:3000->3000/tcp                             grafana
oofnikj commented 3 years ago

Oh, I just realized an omission - Docker assumes traffic is TCP unless you specify otherwise. In the docker-compose config, make sure the second InfluxDB port is specified like this: - "25826:25826/udp"

Your output looks good, except for the lack of measurements. I see that tcpdump reverse-resolves the IP address for your InfluxDB container as rpi-influxDB.lan. If you run tcpdump with the -n option (no reverse lookup) what IP does it show? I'm curious where this reverse record is coming from.

darrepac commented 3 years ago

About tcpdump, it is the name I gave to this pi, so should be normal. Yet here is the output with the -n flag:

pi@raspberrypi:~/Openwrt_graph/docker-openwrt/monitoring $ sudo tcpdump -n -i eth0 udp port 25826
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:04:10.422695 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1329
11:04:10.426034 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1331
11:04:10.436084 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1323
11:04:10.439288 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1285
11:04:10.442557 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1298
11:04:10.445334 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1334
11:04:10.448102 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1328
11:04:10.450942 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1313
11:04:10.459068 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1338
11:04:10.462111 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1316
11:04:10.475503 IP 192.168.0.1.53354 > 192.168.0.53.25826: UDP, length 1312
^C
11 packets captured
11 packets received by filter
0 packets dropped by kernel

About the change in yaml, does the following command is enough to take it into account: sudo docker-compose restart influxdb If yes, I still have nothing in grafana and no measurements in the database

oofnikj commented 3 years ago

Restarting the container after changing the network config isn't enough, you must delete it and recreate it like this:

$ sudo docker-compose rm --stop --force influxdb
$ sudo docker-compose up -d

Note that since we store the data in a volume, deleting and recreating the InfluxDB container like this in the future will not delete any data.

darrepac commented 3 years ago

That was my doubt. now it works!

Except the network graph that is void and wifi signal quality and DNS request, may-be some collectd-mod missing??

oofnikj commented 3 years ago

Great.

I think for those you need collectd-mod-dns and collectd-mod-iwinfo to be installed.

darrepac commented 3 years ago

strange for wifi signal as I have already iwinfo, don't care that much about DNS, but I am worried with this left one being void and should be linked with iptmon isn't it: image

Second question, I want to change something in influxdb.conf because I don't the quantity of data to grow without limit. So I plan to put retention-policy = "3w" in collectd database. Does the following command is ok to take the new influxdb.conf into account?

$ sudo docker-compose rm --stop --force influxdb
$ sudo docker-compose up -d
oofnikj commented 3 years ago

Re: "Network Traffic" panel you need to enable the Interface plugin: Screen Shot 2021-03-03 at 16 38 10

Retention policy config sounds reasonable to me. I don't have mine set, but if I query it it shows 168h (1 week), which I guess is the default.

That command should work to reload the config.

darrepac commented 3 years ago

yes but the network traffic was already enable, that's why I am surprised, and that's why I put the right part of the screenshot...

If retention default is 1 week, I will let it as it is

oofnikj commented 3 years ago

Oh I see, didn't notice that there are values there. Open a shell to your InfluxDB database:

$ sudo docker exec -it influxdb influx -database collectd

Does this query produce results?

> SELECT * FROM interface_rx WHERE instance = 'br-lan' GROUP BY type ORDER BY time DESC LIMIT 3;
darrepac commented 3 years ago

pi@raspberrypi:~/Openwrt_graph/docker-openwrt/monitoring $  sudo docker exec -it influxdb influx -database collectd
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> SELECT * FROM interface_rx WHERE instance = 'br-lan' GROUP BY type ORDER BY time DESC LIMIT 3;
name: interface_rx
tags: type=if_packets
time                host    instance value
----                ----    -------- -----
1614785671477821192 OpenWrt br-lan   4934887
1614785641441521540 OpenWrt br-lan   4932913
1614785611496445844 OpenWrt br-lan   4930583

name: interface_rx
tags: type=if_octets
time                host    instance value
----                ----    -------- -----
1614785671477835345 OpenWrt br-lan   1343578889
1614785641441532385 OpenWrt br-lan   1343137500
1614785611496459975 OpenWrt br-lan   1342531200

name: interface_rx
tags: type=if_errors
time                host    instance value
----                ----    -------- -----
1614785671477846509 OpenWrt br-lan   0
1614785641441543246 OpenWrt br-lan   0
1614785611496470977 OpenWrt br-lan   0

name: interface_rx
tags: type=if_dropped
time                host    instance value
----                ----    -------- -----
1614785671477858226 OpenWrt br-lan   0
1614785641441553638 OpenWrt br-lan   0
1614785611496482269 OpenWrt br-lan   0
>
oofnikj commented 3 years ago

OK, so the data is all there. It seems to be an issue with the Grafana configuration. Are your dashboard variables configured correctly? Screen Shot 2021-03-03 at 17 40 25

darrepac commented 3 years ago

I am using the dashboard from the ID, it should be ok: image

darrepac commented 3 years ago

You know what, I just made a copy of the dashboard in order to be able to modify it...and by magic, everything appear! Even wifi signal quality stuff! image

darrepac commented 3 years ago

I think you are able to close this issue Thank you for the very great support!!

oofnikj commented 3 years ago

Excellent, glad to help. Sometimes a little poking and prodding is all you need to get things to work.

One note about modifying the dashboard - since it's a provisioned dashboard, you will need to export the dashboard JSON and save it to a file to persist the changes.

darrepac commented 3 years ago

Thanks again, I wrote a little article to promote your scripts (in french): https://blocnote360.wordpress.com/2021/03/07/monitorer-le-trafic-internet-dans-sa-maison-avec-openwrt/

oofnikj commented 3 years ago

Brilliant! Thanks very much!