tsunghanliu / adsb-box.snap

a snap to build a ADS-B receiver station
28 stars 4 forks source link

Graphs not being generated correctly #52

Closed satmandu closed 2 years ago

satmandu commented 2 years ago

I see this on a Raspiberry Pi 4B running Raspberry Pi OS 64-bit:

image

Are there specific logging settings I need enabled to get these to show up properly?

satmandu commented 2 years ago

Machine context:

 ,g$$$$$$$$$$$$$$$P.       ---------------- 
  ,g$$P"        """Y$$.".     OS: Debian GNU/Linux 11 (bullseye) aarch64 
 ,$$P'              `$$$.     Host: Raspberry Pi 4 Model B Rev 1.1 
',$$P       ,ggs.     `$$b:   Kernel: 5.15.56-v8+ 
`d$$'     ,$P"'   .    $$$    Uptime: 1 day, 20 hours, 5 mins 
 $$P      d$'     ,    $$P    Packages: 1411 (dpkg), 5 (snap) 
 $$:      $$.   -    ,d$$'    Shell: bash 5.1.4 
 $$;      Y$b._   _,d$P'      Resolution: 2304x1440 
 Y$$.    `.`"Y$$$$P"'         WM: Mutter 
 `$$b      "-.__              WM Theme: Adwaita 
  `Y$$                        Theme: PiXflat [GTK3] 
   `Y$$.                      Icons: PiXflat [GTK3] 
     `$$b.                    Terminal: /dev/pts/0 
       `Y$$b.                 CPU: ARM Cortex-A72 (4) @ 1.500GHz 
          `"Y$b._             Memory: 322MiB / 3793MiB 
              `"""
tsunghanliu commented 2 years ago

hmm, I don't expect any additional setting to enable default graphs. Could you please run the following commands and provide the output?

$ sudo snap logs -n 200 adsb-box.collectd
$ sudo snap run --shell adsb-box.graphs-gend
$ cd $SNAP
$ env XDG_DATA_HOME="$SNAP/usr/share" FONTCONFIG_PATH=$SNAP/etc/fonts bash -x usr/share/graphs1090/graphs1090.sh 24h 0.7
satmandu commented 2 years ago

requested-logs.txt

satmandu commented 2 years ago

Now I see this:

image

tsunghanliu commented 2 years ago

Are all graphs generated correctly now? I'm not sure if it's just because the graphs were not ready at that moment.

Another points is that the /var/snap/adsb-box/current/disk.conf doesn't have right settings and causes the disk graphs wrong. You can change the content to the following and see if this helps.

DISK="mmcblk0"
MOUNTPOINT="/var/lib/snapd/hostfs/root"
satmandu commented 2 years ago

Here is what I see now after modifying that config file thus, since my boot disk is an external USB volume:

DISK="sda"
MOUNTPOINT="/var/lib/snapd/hostfs/root"

(This is after rerunning the commands in the previous comment.) Screenshot_20220807-090428.png

tsunghanliu commented 2 years ago

I forgot to mention that you need to restart the snap (at lease the collectd & graphs-gend) to apply the change.

$ sudo snap restart adsb-box.collectd
$ sudo snap restart adsb-box.graphs-gend

If you didn't do these, please try again.

also, could you please /var/snap/adsb-box/current/collectd.log and output of `find /var/snap/adsb-box/common/collectd/rrd/? I am wondering if collectd gets data correctly or graphs-gend can't find the DB.

satmandu commented 2 years ago
ls -aFl /var/snap/adsb-box/current/collectd/
total 28572
drwxr-xr-x 2 root root     4096 Aug  7  2019 ./
drwxr-xr-x 9 root root     4096 Aug  7 09:01 ../
-rw------- 1 root root     4703 Aug  8 09:40 collectd.conf
-rw-r--r-- 1 root root 29236382 Aug  8 10:27 collectd.log

collectd.log.gz

find /var/snap/adsb-box/common/collectd/rrd/
/var/snap/adsb-box/common/collectd/rrd/
/var/snap/adsb-box/common/collectd/rrd/localhost-v1
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-wlan0
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-wlan0/if_octets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-wlan0/if_packets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-wlan0/if_errors.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-user.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-nice.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-softirq.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-interrupt.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-wait.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-idle.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-system.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/aggregation-cpu-average/cpu-steal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory/memory-slab_recl.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory/memory-slab_unrecl.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory/memory-free.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory/memory-used.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory/memory-cached.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/memory/memory-buffered.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_messages-local_accepted.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_messages-strong_signals.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_cpu-reader.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_cpu-demod.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_messages-remote_accepted_0.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_aircraft-recent.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_dbfs-signal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_dbfs-noise.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_tracks-all.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_messages-positions.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_tracks-single_message.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_messages-local_accepted_0.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_messages-remote_accepted.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_cpu-background.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_dbfs-peak_signal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_range-max_range.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/dump1090-localhost/dump1090_mlat-recent.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/table-localhost
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/table-localhost/gauge-cpu_temp.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-eth0
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-eth0/if_octets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-eth0/if_packets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost-v1/interface-eth0/if_errors.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-wlan0
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-wlan0/if_dropped.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-wlan0/if_octets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-wlan0/if_packets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-wlan0/if_errors.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-user.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-nice.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-softirq.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-interrupt.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-wait.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-idle.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-system.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/aggregation-cpu-average/cpu-steal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-slab_recl.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-slab_unrecl.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-free.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-buffers.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-used.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-cached.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/system_stats/memory-buffered.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/memory
/var/snap/adsb-box/common/collectd/rrd/localhost/memory/memory-slab_recl.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/memory/memory-slab_unrecl.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/memory/memory-free.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/memory/memory-used.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/memory/memory-cached.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/memory/memory-buffered.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_messages-local_accepted.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-min_signal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_messages-strong_signals.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_cpu-reader.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_cpu-demod.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_messages-remote_accepted_0.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_range-median.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_aircraft-recent.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-signal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_tisb-recent.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_range-minimum.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-noise.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_gps-recent.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_tracks-all.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_messages-positions.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_tracks-single_message.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-median.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-NaN.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-quart3.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_messages-local_accepted_0.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_messages-remote_accepted.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_cpu-background.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_range-quart3.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-quart1.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_range-quart1.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_dbfs-peak_signal.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_range-max_range.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_mlat-recent.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/table-localhost
/var/snap/adsb-box/common/collectd/rrd/localhost/table-localhost/gauge-cpu_temp.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-eth0
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-eth0/if_dropped.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-eth0/if_octets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-eth0/if_packets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/interface-eth0/if_errors.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/disk-sda
/var/snap/adsb-box/common/collectd/rrd/localhost/disk-sda/disk_octets.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/disk-sda/disk_ops.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/disk-sda/disk_io_time.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/disk-sda/disk_time.rrd
/var/snap/adsb-box/common/collectd/rrd/localhost/disk-sda/disk_merged.rrd

No changes seen in graph output: image

satmandu commented 2 years ago

(I do not see any changes in more recent versions of collectd which might change things on a Raspberry PI.)

tsunghanliu commented 2 years ago

The adsb-box has its own collectd, so the version of collectd on Debian shouldn't affect the snap.

I revisited your collectd.log and found that there are many rrdtool update errors since June:

[2022-06-08 11:16:16] rrdtool plugin: rrd_update_r (/tmp/rrd/localhost/dump1090-localhost/dump1090_cpu-demod.rrd) failed: mmaping file '/tmp/rrd/localhost/dump1090-localhost/dump1090_cpu-demod.rrd': Invalid argument

I'm not sure if they are caused by data corruptions. To recovery this, you can try these steps:

  1. sudo snap stop adsb-box
  2. back up /var/snap/adsb-box/common/collectd/rrd/localhost
  3. remove /var/snap/adsb-box/common/collectd/rrd/localhost/dump1090-localhost/dump1090_cpu-demod.rrd
  4. sudo snap start adsb-box and check the graphs

You may run fsck to check the disk as well.

satmandu commented 2 years ago

That fixed it!