search

lukdwo / OpenWRT-collectd-MQTT-HA

OpenWRT Collectd to MQTT with Homeassistant Template
62 stars 10 forks source link

No data recorded if main folder is moved to `/etc/collectd` #4

Closed yousaf465 closed 7 months ago

yousaf465 commented 10 months ago

after changing the main folder to /etc/collectd no data is being recorded.

image

my setting for collectd

image

there was no /etc/collectd/ folder, had to to create a new one. only collectd.conf file was present. with the following content

image

BaseDir "/etc/collectd"
Include "/etc/collectd/conf.d/*.conf"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/lib/collectd"
TypesDB "/usr/share/collectd/types.db"
Interval 30
ReadThreads 2

LoadPlugin cpu
<Plugin cpu>
    ValuesPercentage true
    ReportByCpu true
    ReportByState true
</Plugin>

LoadPlugin interface
<Plugin interface>
    IgnoreSelected false
    Interface "br-lan"
</Plugin>

LoadPlugin iwinfo

LoadPlugin load

LoadPlugin memory
<Plugin memory>
    ValuesPercentage false
    ValuesAbsolute true
</Plugin>

LoadPlugin rrdtool
<Plugin rrdtool>
    DataDir "/tmp/rrd"
    RRARows 288
    RRASingle true
    RRATimespan 7200
    RRATimespan 86400
    RRATimespan 604800
    RRATimespan 2678400
    RRATimespan 31622400
</Plugin>

also created this mqtt.conf file in /etc/collectd/conf.d with the following content

image

but it didn't connect with the MQTT broker

image

auanasgheps commented 10 months ago

Hi, this might be an oversight by me.

Try to put restore it to /var/run/collectd and let me know if it works.

yousaf465 commented 10 months ago

/var/run/collectd

after changing back to this directory I do have the graphs but no data

image

image

yousaf465 commented 10 months ago

removing collectd and reinstalling it via SSH came up with this error

opkg remove collectd
No packages removed.
Collected errors:
 * print_dependents_warning: Package collectd is depended upon by packages:
 * print_dependents_warning:    luci-app-statistics
 * print_dependents_warning:    collectd-mod-interface
 * print_dependents_warning:    collectd-mod-network
 * print_dependents_warning:    collectd-mod-mqtt
 * print_dependents_warning:    collectd-mod-iwinfo
 * print_dependents_warning:    collectd-mod-load
 * print_dependents_warning:    collectd-mod-rrdtool
 * print_dependents_warning:    collectd-mod-memory
 * print_dependents_warning:    collectd-mod-cpu
 * print_dependents_warning: These might cease to work if package collectd is re                                                          moved.

 * print_dependents_warning: Force removal of this package with --force-depends.
 * print_dependents_warning: Force removal of this package and its dependents
 * print_dependents_warning: with --force-removal-of-dependent-packages.
root@OpenWrt:~# opkg remove collectd ^C

root@OpenWrt:~# opkg remove collectd --force-removal-of-dependent-packages
Removing package luci-app-statistics from root...
Not deleting modified conffile /etc/config/luci_statistics.
Removing package collectd-mod-interface from root...
Removing package collectd-mod-network from root...
Removing package collectd-mod-mqtt from root...
Removing package collectd-mod-iwinfo from root...
Removing package collectd-mod-load from root...
Removing package collectd-mod-rrdtool from root...
Removing package collectd-mod-memory from root...
Removing package collectd-mod-cpu from root...
Removing package collectd from root...
Command failed: Not found
root@OpenWrt:~# opkg install luci-app-statistics
Installing luci-app-statistics (git-23.315.63824-5a81162) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/lu                                                          ci/luci-app-statistics_git-23.315.63824-5a81162_all.ipk
Installing collectd (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-rrdtool (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-rrdtool_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-iwinfo (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-iwinfo_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-cpu (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-cpu_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-memory (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-memory_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-interface (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-interface_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-load (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-load_5.12.0-49_mips_24kc.ipk
Installing collectd-mod-network (5.12.0-49) to root...
Downloading https://downloads.openwrt.org/releases/23.05.2/packages/mips_24kc/pa                                                          ckages/collectd-mod-network_5.12.0-49_mips_24kc.ipk
Configuring collectd.
Configuring collectd-mod-rrdtool.
Configuring collectd-mod-iwinfo.
Configuring collectd-mod-cpu.
Configuring collectd-mod-memory.
Configuring collectd-mod-interface.
Configuring collectd-mod-load.
Configuring collectd-mod-network.
Configuring luci-app-statistics.
Collected errors:
 * resolve_conffiles: Existing conffile /etc/config/luci_statistics is different                                                           from the conffile in the new package. The new conffile will be placed at /etc/c                                                          onfig/luci_statistics-opkg.
root@OpenWrt:~#

fixed it by renmaing the new luci_statistics-opkg to luci_statistics.

got data, now need to send it over to HA. image image

auanasgheps commented 10 months ago

Ok please let me know if the rest of the setup goes well. I'll delete the part of the base directory.

yousaf465 commented 10 months ago

Ok please let me know if the rest of the setup goes well. I'll delete the part of the base directory.

image

image

no MQTT connection with HA. even though data is there

image

auanasgheps commented 10 months ago

Can you see any error in OpenWRT logs? Also, in MQTT Explorer can you login successfully using the credentials set in OpenWRT for MQTT?

yousaf465 commented 10 months ago

Can you see any error in OpenWRT logs? Also, in MQTT Explorer can you login successfully using the credentials set in OpenWRT for MQTT?

this is with same login as used in mqtt.conf in OpenWrt image image

No mention of MQTT in OpenWrt logs image

auanasgheps commented 10 months ago

Are you sure the MQTT part is fine? Because that's what it's not working right now. Is the MQTT plugin installed and collectd restarted?

yousaf465 commented 10 months ago

Are you sure the MQTT part is fine? Because that's what it's not working right now. Is the MQTT plugin installed and collectd restarted?

 opkg install collectd-mod-mqtt
Package collectd-mod-mqtt (5.12.0-49) installed in root is up to date.

service collectd restart

logs from OpenWrt

ed Dec  6 15:50:00 2023 cron.err crond[1476]: USER root pid 9200 cmd arp-scan -qxdlN -I br-lan
Wed Dec  6 15:51:42 2023 daemon.err collectd[1707]: Exiting normally.
Wed Dec  6 15:51:42 2023 daemon.err collectd[1707]: collectd: Stopping 2 read threads.
Wed Dec  6 15:51:42 2023 daemon.err collectd[1707]: collectd: Stopping 5 write threads.
Wed Dec  6 15:51:42 2023 daemon.err collectd[1707]: rrdtool plugin: Shutting down the queue thread.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: plugin_load: plugin "cpu" successfully loaded.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: plugin_load: plugin "interface" successfully loaded.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: plugin_load: plugin "iwinfo" successfully loaded.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: plugin_load: plugin "load" successfully loaded.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: plugin_load: plugin "memory" successfully loaded.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: plugin_load: plugin "rrdtool" successfully loaded.
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: rrdtool plugin: RRASingle = true: creating only AVERAGE RRAs
Wed Dec  6 15:51:42 2023 daemon.err collectd[9273]: Initialization complete, entering read-loop.

how to check MQTT is working on router or not? image

MQTT config

image

auanasgheps commented 9 months ago

Apologies for my late response but I don't have much free time right now. I think that collect.d is not loading the mqtt plugin, but I don't know what else needs to be checked. The file is in the right place - Directory for sub-configurations is right!

Maybe you can try to create the file from the terminal itself and not via WinSCP. Also, delete the comments #replace this.... Maybe these are interfering.

lukdwo commented 9 months ago

Hi, You should get listed below line in logs from OpenWrt: plugin load: plugin "mqtt" successfully loaded.

After you reinstall your collectd probably collectd-mod-mqtt has been removed to. Check: opkg list-installed | grep mqtt It should list one package.

yousaf465 commented 9 months ago

Hi, You should get listed below line in logs from OpenWrt: plugin load: plugin "mqtt" successfully loaded.

After you reinstall your collectd probably collectd-mod-mqtt has been removed to. Check: opkg list-installed | grep mqtt It should list one package.

opkg list-installed | grep mqtt
baresip-mod-mqtt - 1.1.0-6
bigclown-mqtt2influxdb - 1.4.0-1
collectd-mod-mqtt - 5.12.0-49
mqttled - 0.1.2-1
python3-paho-mqtt - 1.6.1-2

this is the output from that command

lukdwo commented 9 months ago

Try create config file from CLI as @auanasgheps mentioned

rm -rf /etc/collectd/
mkdir -p /etc/collectd/conf.d
touch /etc/collectd/conf.d/mqtt.conf
vi /etc/collectd/conf.d/mqtt.conf

Reload service: service collectd reload Past result: logread -e mqtt or logread -e collectd

yousaf465 commented 9 months ago

logread -e mqtt

Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,447 INFO:                                                                                                                                                              MQTT broker host: 192.168.1.1, port: 1883, use tls: False
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,459 WARNI                                                                                                                                                             NG: mt76-phy0 not a valid led name
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,461 WARNI                                                                                                                                                             NG: rt2800soc-phy1::assoc not a valid led name
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,464 WARNI                                                                                                                                                             NG: rt2800soc-phy1::quality not a valid led name
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,467 WARNI                                                                                                                                                             NG: rt2800soc-phy1::radio not a valid led name
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,483 INFO:                                                                                                                                                              LED: green-wlan2g subscribed to OpenWRTleds/CustomDevice/green:wlan2g/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,494 INFO:                                                                                                                                                              LED: ath9k-phy0 subscribed to OpenWRTleds/CustomDevice/ath9k-phy0/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,501 INFO:                                                                                                                                                              LED: green-wan subscribed to OpenWRTleds/CustomDevice/green:wan/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,509 INFO:                                                                                                                                                              LED: amber-power subscribed to OpenWRTleds/CustomDevice/amber:power/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,516 INFO:                                                                                                                                                              LED: green-usb subscribed to OpenWRTleds/CustomDevice/green:usb/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,523 INFO:                                                                                                                                                              LED: green-power subscribed to OpenWRTleds/CustomDevice/green:power/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,530 INFO:                                                                                                                                                              LED: blue-wlan5g subscribed to OpenWRTleds/CustomDevice/blue:wlan5g/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,537 INFO:                                                                                                                                                              LED: amber-wan subscribed to OpenWRTleds/CustomDevice/amber:wan/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,544 INFO:                                                                                                                                                              LED: green-wps subscribed to OpenWRTleds/CustomDevice/green:wps/set
Thu Dec 21 22:01:04 2023 daemon.err mqttled[2611]: 2023-12-21 22:01:04,550 INFO:                                                                                                                                                              MQTT LED Control Started
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]: Traceback (most recent call l                                                                                                                                                             ast):
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/bin/mqttled", li                                                                                                                                                             ne 8, in <module>
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:     sys.exit(run())
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:              ^^^^^
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ite-packages/mqttled/__main__.py", line 28, in run
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ite-packages/mqttled/ledHost.py", line 129, in run
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ite-packages/paho/mqtt/client.py", line 1742, in loop_forever
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ite-packages/paho/mqtt/client.py", line 1044, in reconnect
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ite-packages/paho/mqtt/client.py", line 3685, in _create_socket_connection
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ocket.py", line 851, in create_connection
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]:   File "/usr/lib/python3.11/s                                                                                                                                                             ocket.py", line 836, in create_connection
Thu Dec 21 22:01:09 2023 daemon.err mqttled[2611]: TimeoutError: timed out
yousaf465 commented 9 months ago

and 

logread -e collectd
Wed Dec 13 18:36:43 2023 daemon.err collectd[1707]: plugin_load: plugin "cpu" successfully loaded.
Wed Dec 13 18:36:44 2023 daemon.err collectd[1707]: plugin_load: plugin "interface" successfully loaded.
Wed Dec 13 18:36:44 2023 daemon.err collectd[1707]: plugin_load: plugin "iwinfo" successfully loaded.
Wed Dec 13 18:36:44 2023 daemon.err collectd[1707]: plugin_load: plugin "load" successfully loaded.
Wed Dec 13 18:36:45 2023 daemon.err collectd[1707]: plugin_load: plugin "memory" successfully loaded.
Wed Dec 13 18:36:45 2023 daemon.err collectd[1707]: plugin_load: plugin "rrdtool" successfully loaded.
Wed Dec 13 18:36:45 2023 daemon.err collectd[1707]: rrdtool plugin: RRASingle = true: creating only AVERAGE RRAs
Wed Dec 13 18:36:45 2023 daemon.err collectd[1707]: Initialization complete, entering read-loop.
Wed Dec 13 18:36:46 2023 daemon.err collectd[1707]: rrdtool plugin: rrd_update_r failed: /tmp/rrd/OpenWrt/interface-br-lan/if_packets.rrd: Not enough arguments
Thu Dec 21 22:01:09 2023 daemon.err collectd[1707]: rrdtool plugin: rrd_update_r failed: /tmp/rrd/OpenWrt/iwinfo-phy1-ap0/signal_power.rrd: opening '/tmp/rrd/OpenWrt/iwinfo-phy1-ap0/signal_power.rrd': No such file or directory
Thu Dec 21 22:01:09 2023 daemon.err collectd[1707]: Sleeping only 2s because the next interval is 703404.074 seconds in the past!
Thu Dec 21 22:54:45 2023 daemon.err collectd[1707]: Exiting normally.
Thu Dec 21 22:54:45 2023 daemon.err collectd[1707]: collectd: Stopping 2 read threads.
Thu Dec 21 22:54:45 2023 daemon.err collectd[1707]: collectd: Stopping 5 write threads.
Thu Dec 21 22:54:45 2023 daemon.err collectd[1707]: rrdtool plugin: Shutting down the queue thread.
Thu Dec 21 22:54:45 2023 daemon.err collectd[3128]: Parse error in file `/tmp/collectd.conf', line 53 near `': syntax error, unexpected $end, expecting EOL
Thu Dec 21 22:54:45 2023 daemon.err collectd[3128]: yyparse returned error #1
Thu Dec 21 22:54:45 2023 daemon.err collectd[3128]: configfile: Cannot read file `/tmp/collectd.conf'.
Thu Dec 21 22:54:45 2023 daemon.err collectd[3128]: Unable to read config file /tmp/collectd.conf.
Thu Dec 21 22:54:45 2023 daemon.err collectd[3128]: Error: Parsing the config file failed!
Thu Dec 21 22:54:50 2023 daemon.err collectd[3154]: Parse error in file `/tmp/collectd.conf', line 53 near `': syntax error, unexpected $end, expecting EOL
Thu Dec 21 22:54:50 2023 daemon.err collectd[3154]: yyparse returned error #1
Thu Dec 21 22:54:50 2023 daemon.err collectd[3154]: configfile: Cannot read file `/tmp/collectd.conf'.
Thu Dec 21 22:54:50 2023 daemon.err collectd[3154]: Unable to read config file /tmp/collectd.conf.
Thu Dec 21 22:54:50 2023 daemon.err collectd[3154]: Error: Parsing the config file failed!
Thu Dec 21 22:54:55 2023 daemon.err collectd[3157]: Parse error in file `/tmp/collectd.conf', line 53 near `': syntax error, unexpected $end, expecting EOL
Thu Dec 21 22:54:55 2023 daemon.err collectd[3157]: yyparse returned error #1
Thu Dec 21 22:54:55 2023 daemon.err collectd[3157]: configfile: Cannot read file `/tmp/collectd.conf'.
Thu Dec 21 22:54:55 2023 daemon.err collectd[3157]: Unable to read config file /tmp/collectd.conf.
Thu Dec 21 22:54:55 2023 daemon.err collectd[3157]: Error: Parsing the config file failed!
Thu Dec 21 22:55:00 2023 daemon.err collectd[3159]: Parse error in file `/tmp/collectd.conf', line 53 near `': syntax error, unexpected $end, expecting EOL
Thu Dec 21 22:55:00 2023 daemon.err collectd[3159]: yyparse returned error #1
Thu Dec 21 22:55:00 2023 daemon.err collectd[3159]: configfile: Cannot read file `/tmp/collectd.conf'.
Thu Dec 21 22:55:00 2023 daemon.err collectd[3159]: Unable to read config file /tmp/collectd.conf.
Thu Dec 21 22:55:00 2023 daemon.err collectd[3159]: Error: Parsing the config file failed!
Thu Dec 21 22:55:05 2023 daemon.err collectd[3161]: Parse error in file `/tmp/collectd.conf', line 53 near `': syntax error, unexpected $end, expecting EOL
Thu Dec 21 22:55:05 2023 daemon.err collectd[3161]: yyparse returned error #1
Thu Dec 21 22:55:05 2023 daemon.err collectd[3161]: configfile: Cannot read file `/tmp/collectd.conf'.
Thu Dec 21 22:55:05 2023 daemon.err collectd[3161]: Unable to read config file /tmp/collectd.conf.
Thu Dec 21 22:55:05 2023 daemon.err collectd[3161]: Error: Parsing the config file failed!
Thu Dec 21 22:55:10 2023 daemon.err collectd[3163]: Parse error in file `/tmp/collectd.conf', line 53 near `': syntax error, unexpected $end, expecting EOL
Thu Dec 21 22:55:10 2023 daemon.err collectd[3163]: yyparse returned error #1
Thu Dec 21 22:55:10 2023 daemon.err collectd[3163]: configfile: Cannot read file `/tmp/collectd.conf'.
Thu Dec 21 22:55:10 2023 daemon.err collectd[3163]: Unable to read config file /tmp/collectd.conf.
Thu Dec 21 22:55:10 2023 daemon.err collectd[3163]: Error: Parsing the config file failed!
Thu Dec 21 22:55:10 2023 daemon.info procd: Instance collectd::instance1 s in a crash loop 6 crashes, 0 seconds since last crash
lukdwo commented 9 months ago

Check ls -la /tmp/collectd.conf My result is: lrwxrwxrwx 1 root root 18 Dec 21 14:46 /tmp/collectd.conf -> /etc/collectd.conf When you get the same result you have to double check your /etc/collectd.conf especially around line 53

yousaf465 commented 9 months ago

ls -la /tmp/collectd.conf

 ls -la /tmp/collectd.conf
lrwxrwxrwx    1 root     root            18 Dec 21 23:36 /tmp/collectd.conf -> /etc/collectd.conf
yousaf465 commented 9 months ago

lrwxrwxrwx 1 root root 18 Dec 21 23:36 /tmp/collectd.conf -> /etc/collectd.conf

line 53 is missing

image

image

file was created via vi editor.

yousaf465 commented 9 months ago

working finally 

Thu Dec 21 23:49:22 2023 daemon.err collectd[3425]: plugin_load: plugin "mqtt" successfully loaded.

image

lukdwo commented 9 months ago

Another way to remove special characters:

vi /etc/collectd/conf.d/mqtt.conf
:e ++ff=unix 
:%s/\r\(\n\)/\1/g
:wq
auanasgheps commented 9 months ago

@yousaf465 I'm happy that you got it working! Can you please clarify what have you changed? I guess are my comments in the mqtt file, please confirm

yousaf465 commented 9 months ago

@yousaf465 I'm happy that you got it working! Can you please clarify what have you changed? I guess are my comments in the mqtt file, please confirm

here is the working file image

yousaf465 commented 9 months ago

Now need to fix the sensors' names and values

image

how can I get correct name for the missing sensors?

lukdwo commented 9 months ago

You have to change all lines in configuration.yaml to your interfaces name. For instance check ifconfig and corect br-lan in line: state_topic: collectd/<Open-WRT-Hostname>/interface-br-lan/if_errors You can check what your router is sending on MQTT Client.

auanasgheps commented 9 months ago

Yeah, every router will have different config names. I will add this to my to-do list and will try to improve the instructions!

Also, some of these sensors like google ping requires additional packages.

yousaf465 commented 9 months ago

image

image

how to name these in the configuration.yaml?

should use cpu-0 as sensor name for CPU?

auanasgheps commented 9 months ago

You have to adjust the topic name, for example the CPU, if it doesn't match the configuration.yaml example.

There's no way to standardise it as every device is different.