search

wiedehopf / tar1090

Provides an improved webinterface for use with ADS-B decoders readsb / dump1090-fa
Other
1.27k stars 237 forks source link

Question : Issue with Enabling Heatmap Option in readsb Configuration #290

Closed LoickFahl closed 7 months ago

LoickFahl commented 7 months ago

Dear GitHub Community,

I am currently facing an issue while attempting to enable the heatmap option in my readsb configuration. I would like to utilize the heatmap feature for tar1090 with readsb, but encountered difficulties when configuring it to store the generated data on a thumb drive.

Here is my current readsb configuration:

# readsb configuration
# This is sourced by /etc/systemd/system/default.target.wants/readsb.service as
# daemon startup configuration.

RECEIVER_OPTIONS="--device SDR000 --device-type rtlsdr --gain 22.9 --ppm 0 --mlat --forward-mlat --gnss"
DECODER_OPTIONS="--lat [REDACTED] --lon [REDACTED] --max-range 1000 --range-outline-hours 730 --write-json-every 1 --write-json-globe-index --write-globe-history /mnt/Logs/readsb/globe_history --heatmap 10"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005"
JSON_OPTIONS="--json-location-accuracy 2 --db-file /usr/local/share/tar1090/aircraft.csv.gz"

/mnt/Logs directory and its contents have permissions set to drwxrwxrwx.

Thanks

wiedehopf commented 7 months ago

You have to use the recommended directory for the webinterface to work without a change in configuration.

LoickFahl commented 7 months ago

With the recommended directory 


RECEIVER_OPTIONS="--device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss"
DECODER_OPTIONS="--lat xxxx --lon xxxx --max-range 1000 --write-json-every 1 --write-json-globe-index --write-globe-history /var/globe_history --heatmap 30"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005"
JSON_OPTIONS="--json-location-accuracy 2 --range-outline-hours 168 --db-file /usr/local/share/tar1090/aircraft.csv.gz"

I do get the files in /var/globe_history but no heatmap data http://192.168.1.xxx/tar1090/?heatmap=200000&realHeat&heatRadius=2

drwxr-xr-x  4 readsb root       4096 Feb 19 11:31 globe_history

pi@pi:/var/globe_history $ ls -l
total 16
drwxr-xr-x 3 readsb nogroup  4096 Feb 19 11:31 2024
drwxr-xr-x 2 readsb nogroup 12288 Feb 19 12:54 internal_state
wiedehopf commented 7 months ago

It will take time to accumulate some data. Also don't know how your webserver is configured.

The heatmap data will be by day ... in the corresponding subfolder.

wiedehopf commented 7 months ago

To be a bit more precise, it will write heatmap data in chunks of 30 min, so after 30 min you should have at least one file.

LoickFahl commented 7 months ago

Thank you for the precision. I do have the files but nothing is showing up. pi@pi:/var/globe_history/2024/02/19 $ ls acas heatmap traces

pi@pi:/var/globe_history/2024/02/19 $ ls -l
total 12
drwxr-xr-x   2 readsb nogroup 4096 Feb 19 11:31 acas
drwxr-xr-x   2 readsb nogroup 4096 Feb 19 12:58 heatmap
drwxr-xr-x 258 readsb nogroup 4096 Feb 19 11:31 traces
pi@pi:/var/globe_history/2024/02/19 $ ls heatmap
20.bin.ttf  22.bin.ttf

Also on the read.md you have (1) --write-json-globe-index --write-globe-history /var/globe_history --heatmap 30 var/globe_history needs to be a directory writeable by the user readsb. sudo mkdir /var/globe_history and sudo chown readsb /var/globe_history are useful for that.

You should also download

wget -O /usr/local/share/tar1090/aircraft.csv.gz https://github.com/wiedehopf/tar1090-db/raw/csv/aircraft.csv.gz

Followed by in the end

(2) var/globe_history needs to be a directory writeable by the user readsb. sudo mkdir /var/globe_history and sudo chown readsb /var/globe_history are useful for that.

Add readsb options:

--heatmap-dir /var/globe_history --heatmap 30

The first one (1)

DECODER_OPTIONS="--lat xxx --lon xxxx --max-range 1000 --write-json-every 1 --write-json-globe-index --write-globe-history /var/globe_history --heatmap 1"

cause this issue

Feb 19 14:00:27 pi systemd[1]: readsb.service: Main process exited, code=killed, status=7/BUS
Feb 19 14:00:27 pi systemd[1]: readsb.service: Failed with result 'signal'.

(2) doesn't causes issues

DECODER_OPTIONS="--lat xxx --lon xxx --max-range 1000 --write-json-every 1 --write-json-globe-index --heatmap-dir /var/globe_history --heatmap 30"

In either case no data is displayed

wiedehopf commented 7 months ago

Possibly the 1 second heatmap interval ... to be honest i'm not very tempted to look into it. Possibly i will.

You can always check what files it's trying to load when you call up the heatmap and then check the log for your webserver. Note it will try to get data for the last 24h so not all files being there is expected in your case.

This is a somewhat niche feature and it it doesn't work for everyone that's possibly just how it's gonna be.

wiedehopf commented 7 months ago

Just to elaborate on chasing down the data not being displayed. In the browser, check the dev console which paths it's trying to load, you can filter the network log by "heatmap" Then check if any of those files it's trying to load are present on the disk.

I don't know how you installed tar1090 and which webserver you're using with it. So that can affect if the files can get loaded or not.

May i suggest --heatmap-dir /var/globe_history --heatmap 5 --json-trace-interval 5

If you don't lower the trace interval the low heatmap interval makes not so much sense. --write-json-globe-index shouldn't be needed just for the heatmap data.

wiedehopf commented 7 months ago

As for checking if any file were written:

find /var/globe_history | grep heat

I can't reproduce your error for --heatmap 1 so you'd have to run gdb probably to narrow down the issue. So ... let's ignore that for now.

Restarting with only heatmap-dir and no --write-state will cause data loss so i'd recommend

--heatmap-dir /var/globe_history --write-state /var/globe_history --heatmap 5 --json-trace-interval 5

Most likely issue right now is the webserver. You could always run it in docker, even has heatmap enabled by default: https://github.com/sdr-enthusiasts/docker-tar1090 I'm not the maintainer of that though.

wiedehopf commented 7 months ago

Just noticed something, this can be pretty transparent: http://192.168.1.xxx/tar1090/?heatmap=200000&realHeat&heatRadius=2

Just try this and see if there is any data: http://192.168.1.xxx/tar1090/?heatmap

LoickFahl commented 7 months ago

Thanks, For tar1090, I used your page, single instance manual install not docker. sudo bash -c "$(wget -nv -O - https://github.com/wiedehopf/tar1090/raw/master/install.sh)"

No change to /usr/local/share/tar1090/html/config.js

#  /etc/default/tar1090

# Time in seconds between snapshots in the track history
INTERVAL=1

# How many snapshots are stored in the track history
HISTORY_SIZE=5000

# history duration computes as HISTORY_SIZE x INTERVAL seconds.

# Change to yes to enable UAT/978 display in tar1090
ENABLE_978=no
# If running dump978-fa on another computer, modify the IP-address as appropriate.
# The URL needs to point at where you would normally find the skyaware978 webinterface
URL_978="http://127.0.0.1/skyaware978"

# 1-9 are valid, lower lvl: less CPU usage, higher level: less network bandwidth used when loading the page
GZIP_LVL=1

# hours of tracks that /?pTracks will show
PTRACKS=168

# no need to change the lines below
CHUNK_SIZE=20
INT_978=1
# disable getting planefinder local client data by default
PF_ENABLE=0
PF_URL="http://127.0.0.1:30053/ajax/aircraft"
COMPRESS_978=""

For the heatmap data only 2 dot have appeared (my setup has an average of 600 messages per seconds with at this time 50+ aircraft visible and tracked on tar1090) So something is not right.

RECEIVER_OPTIONS="--device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss"
DECODER_OPTIONS="--lat xxx --lon xxx --max-range 1000 --write-json-every 1"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005"
JSON_OPTIONS="--json-location-accuracy 2 --range-outline-hours 168 --write-json /run/readsb/ --write-json-globe-index --heatmap-dir /var/globe_history --heatmap 30"

I will test with and report in a few hours.

Without --write-state /var/globe_history or --heatmap-dir /var/globe_history as with both present it fails

Feb 19 20:04:14 pi systemd[1]: Stopped readsb ADS-B receiver.
Feb 19 20:04:14 pi systemd[1]: Started readsb ADS-B receiver.
Feb 19 20:04:14 pi readsb[14190]: invoked by: /usr/bin/readsb --device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss --lat xxx --lon xxx --max-range 1000 --net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168 --heatmap-dir /var/globe_history --write-state /var/globe_history --heatmap 5 --write-json /run/readsb --quiet
Feb 19 20:04:14 pi readsb[14190]: [2024-02-19 20:04:14.973 CET] readsb starting up.
Feb 19 20:04:14 pi readsb[14190]: readsb version: 3.14.1610 wiedehopf git: 11bf36c (committed: Sun Feb 18 18:37:48 2024 0100)
Feb 19 20:04:14 pi readsb[14190]: Using lat:   xxxxx, lon:    xxx
Feb 19 20:04:14 pi readsb[14190]: loading state .....
Feb 19 20:04:15 pi readsb[14190]: actual range outline, read bytes: 368644
Feb 19 20:04:15 pi readsb[14190]:  .......... done, loaded 96 aircraft in 0.037 seconds!
Feb 19 20:04:15 pi readsb[14190]: aircraft table fill: 0.0
Feb 19 20:04:15 pi readsb[14190]: 30002: Raw TCP output port
Feb 19 20:04:15 pi readsb[14190]: 30005: Beast TCP output port
Feb 19 20:04:15 pi readsb[14190]: 30003: SBS TCP output ALL port
Feb 19 20:04:15 pi readsb[14190]: 30001: Raw TCP input port
Feb 19 20:04:15 pi readsb[14190]: 30004: Beast TCP input port
Feb 19 20:04:15 pi readsb[14190]: 30104: Beast TCP input port
Feb 19 20:04:15 pi readsb[14190]: rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
Feb 19 20:04:15 pi readsb[14190]: Found Rafael Micro R820T tuner
Feb 19 20:04:15 pi readsb[14190]: rtlsdr: tuner gain set to 25.4 dB
Feb 19 20:04:15 pi readsb[14190]: Allocating 16 zero-copy buffers
**Feb 19 20:04:18 pi systemd[1]: readsb.service: Main process exited, code=killed, status=7/BUS
Feb 19 20:04:18 pi systemd[1]: readsb.service: Failed with result 'signal'.**
Feb 19 20:04:36 pi systemd[1]: Stopped readsb ADS-B receiver.
Feb 19 20:04:36 pi systemd[1]: Started readsb ADS-B receiver.
RECEIVER_OPTIONS="--device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss"
DECODER_OPTIONS="--lat xxx --lon xxx --max-range 1000 --write-json-every 1"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005"
JSON_OPTIONS="--json-location-accuracy 2 --write-json-every 1 --write-json-every 1 --range-outline-hours 168 --write-json-globe-index --write-json /run/readsb/ --write-state /var/globe_history --heatmap 5"

Additional info

pi@pi:/var/globe_history/2024/02/19/heatmap $ ls
20.bin.ttf  22.bin.ttf  24.bin.ttf  35.bin.ttf  36.bin.ttf
pi@pi:/var/globe_history/2024/02/19/heatmap $ ls -l
total 28
-rw-r--r-- 1 readsb nogroup  977 Feb 19 11:58 20.bin.ttf
-rw-r--r-- 1 readsb nogroup  398 Feb 19 12:58 22.bin.ttf
-rw-r--r-- 1 readsb nogroup 8424 Feb 19 13:58 24.bin.ttf
-rw-r--r-- 1 readsb nogroup  396 Feb 19 19:28 35.bin.ttf
-rw-r--r-- 1 readsb nogroup 1916 Feb 19 19:50 36.bin.ttf
wiedehopf commented 7 months ago

Huh seems it didn't write all the files. Possibly it just crashed / stopped working in between?

So i don't know your command line knowledge so ... feel free to ask for more detail.

sudo apt install gdb

When gdb is installed it would be good to run readsb from the command line in gdb, easiest to do that by using this script: https://github.com/wiedehopf/readsb/blob/dev/gdb.sh

You could just clone the readsb repo or download it separately. This will stop the normal readsb service and run it on the command line.

You can just Ctrl-C to stop it and change /etc/default/readsb to change parameters. Provoking the BUS error would be nice as ... it's reasonable to assume it's related to the issues.

Oh also i'm curious

uname -a
cat /etc/os-release
LoickFahl commented 7 months ago

Thank you for your help. Yes, i've tested a few combinaisons, some tended to crash after launch some after a few minutes.

If it helps the /?pTracks works fine

/?replay not so much 

Problem fetching data from the server:
HTTP error, status = 404 --> No data for this timestamp

For the Os

pi@pi:~ $ uname -a
cat /etc/os-release
Linux pi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

And I've launched the script 

pi@pi:~ $ sudo chmod +777 gdb.sh
pi@pi:~ $ sudo ./gdb.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[Detaching after fork from child process 10366]
invoked by: /usr/bin/readsb --quiet --device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss --lat xxx --lon xxx --max-range 1000 --net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168 --write-json /run/readsb/ --write-json-globe-index --write-state /var/globe_history --heatmap 5 --json-trace-interval 5 --write-json /run/readsb
[2024-02-19 22:39:10.385 CET] readsb starting up.
readsb version: 3.14.1610 wiedehopf git: 11bf36c (committed: Sun Feb 18 18:37:48 2024 0100)
Heatmap requires globe history dir or heatmap dir to be set, disabling heatmap!
Using lat:   xxx, lon:    xxx
loading state .....
actual range outline, read bytes: 368644
 .......... done, loaded 299 aircraft in 0.055 seconds!
aircraft table fill: 0.0
30002: Raw TCP output port
30005: Beast TCP output port
30003: SBS TCP output ALL port
30001: Raw TCP input port
30004: Beast TCP input port
30104: Beast TCP input port
rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
Found Rafael Micro R820T tuner
rtlsdr: tuner gain set to 25.4 dB
Allocating 16 zero-copy buffers
Wrote live traces for aircraft active within the last 15 minutes. This took 4.0 seconds (roughly 0 minutes).
Wrote live traces for aircraft active within the last 24 hours. This took 4.0 seconds (roughly 0 minutes).

Heatmap requires globe history dir or heatmap dir to be set, disabling heatmap! This seems to be the issue

I've changed
JSON_OPTIONS="--json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168 --write-json-globe-index --heatmap-dir /var/globe_history --heatmap 5 --json-trace-interval 5"

pi@pi:~ $ sudo ./gdb.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[Detaching after fork from child process 1958]
invoked by: /usr/bin/readsb --quiet --device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss --lat xxx --lon xxx --max-range 1000 --net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168  --write-json-globe-index --heatmap-dir /var/globe_history --heatmap 5 --json-trace-interval 5 --write-json /run/readsb
[2024-02-19 22:50:49.695 CET] readsb starting up.
readsb version: 3.14.1610 wiedehopf git: 11bf36c (committed: Sun Feb 18 18:37:48 2024 0100)
Using lat:   xxx, lon:    xxx
30002: Raw TCP output port
30005: Beast TCP output port
30003: SBS TCP output ALL port
30001: Raw TCP input port
30004: Beast TCP input port
30104: Beast TCP input port
rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
Found Rafael Micro R820T tuner
rtlsdr: tuner gain set to 25.4 dB
Allocating 16 zero-copy buffers
Wrote live traces for aircraft active within the last 15 minutes. This took 4.1 seconds (roughly 0 minutes).
Wrote live traces for aircraft active within the last 24 hours. This took 4.0 seconds (roughly 0 minutes).

Let's see if it improve things .

pi@pi:/var/globe_history/2024/02/19/heatmap $ ls -l
total 28
-rw-r--r-- 1 readsb nogroup  977 Feb 19 11:58 20.bin.ttf
-rw-r--r-- 1 readsb nogroup  398 Feb 19 12:58 22.bin.ttf
-rw-r--r-- 1 readsb nogroup 8424 Feb 19 13:58 24.bin.ttf
-rw-r--r-- 1 readsb nogroup  396 Feb 19 19:28 35.bin.ttf
-rw-r--r-- 1 readsb nogroup 1916 Feb 19 19:50 36.bin.ttf
wiedehopf commented 7 months ago

Well possibly but not for the bus error.

--heatmap-dir /var/globe_history --write-state /var/globe_history

This is still the combo you want and if it causes a BUS error now ... we might see what's wrong.

--write-json-globe-index changes some stuff most people don't want to change. Unless you want traces as well but then it's just --write-globe-history with that

Possibly the gdb script isn't working well with the sudo ... it's kinda annoying to run this stuff and not screw up permissions in /var/globe_history I've fixed the gdb.sh script can you re-download and use the new version? Otherwise it probably won't give any useful info when it crashes :)

LoickFahl commented 7 months ago

It crashed Ok, with the new script

pi@pi:~ $ sudo nano gdb.sh
pi@pi:~ $ sudo chmod +777 gdb.sh
pi@pi:~ $ ./gdb.sh
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to stop 'readsb.service'.
Authenticating as: ,,, (pi)
Password: 
==== AUTHENTICATION COMPLETE ===
mkdir: cannot create directory ‘/run/readsb’: Permission denied
chown: cannot access '/run/readsb': No such file or directory
runuser: may not be used by non-root users
pi@pi:~ $ sudo ./gdb.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/bin/readsb --quiet --device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss --lat 000 --lon 000 --max-range 1000 --net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-            port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168 --write-json-globe-index --heatmap-dir /var/globe_history --write-state /var/globe_history --heatmap 5 --json-trace-interval 5 --write-json /run/readsb
[2024-02-19 23:39:46.162 CET] readsb starting up.
readsb version: 3.14.1610 wiedehopf git: 11bf36c (committed: Sun Feb 18 18:37:48 2024 0100)
Using lat:   000, lon:    000
[New Thread 0xf7bb9440 (LWP 5705)]
[New Thread 0xf73b8440 (LWP 5706)]
[New Thread 0xf6bb7440 (LWP 5707)]
[New Thread 0xf63b6440 (LWP 5708)]
loading state .....
[New Thread 0xf58a9440 (LWP 5709)]
[New Thread 0xf50a8440 (LWP 5710)]
[New Thread 0xf48a7440 (LWP 5711)]
[New Thread 0xf40a6440 (LWP 5712)]
actual range outline, read bytes: 368644
[Thread 0xf50a8440 (LWP 5710) exited]
[Thread 0xf58a9440 (LWP 5709) exited]
[Thread 0xf48a7440 (LWP 5711) exited]
[Thread 0xf40a6440 (LWP 5712) exited]
 .......... done, loaded 308 aircraft in 0.050 seconds!
aircraft table fill: 0.0
[New Thread 0xf40a6440 (LWP 5713)]
[New Thread 0xf48a7440 (LWP 5714)]
[New Thread 0xf58a9440 (LWP 5716)]
[New Thread 0xf50a8440 (LWP 5715)]
[New Thread 0xf30ff440 (LWP 5717)]
30002: Raw TCP output port
30005: Beast TCP output port
30003: SBS TCP output ALL port
30001: Raw TCP input port
30004: Beast TCP input port
30104: Beast TCP input port
[New Thread 0xf28fe440 (LWP 5718)]
[New Thread 0xf20fd440 (LWP 5719)]
[New Thread 0xf16fc440 (LWP 5720)]
[New Thread 0xf0cff440 (LWP 5721)]
[Thread 0xf50a8440 (LWP 5715) exited]
[New Thread 0xf50a8440 (LWP 5722)]
[New Thread 0xf02ff440 (LWP 5723)]
[New Thread 0xefafe440 (LWP 5724)]
[New Thread 0xef2fd440 (LWP 5725)]
[Thread 0xf50a8440 (LWP 5722) exited]
[New Thread 0xf50a8440 (LWP 5726)]
[Thread 0xf50a8440 (LWP 5726) exited]
[New Thread 0xf50a8440 (LWP 5727)]
[Thread 0xf50a8440 (LWP 5727) exited]
[New Thread 0xf50a8440 (LWP 5728)]
rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
[New Thread 0xf50a8440 (LWP 5734)]
[Thread 0xf50a8440 (LWP 5728) exited]
Found Rafael Micro R820T tuner
rtlsdr: tuner gain set to 25.4 dB
Allocating 16 zero-copy buffers
Wrote live traces for aircraft active within the last 15 minutes. This took 4.0 seconds (roughly 0 minutes).
Wrote live traces for aircraft active within the last 24 hours. This took 4.0 seconds (roughly 0 minutes).

With the updated Json

JSON_OPTIONS="--json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168 --write-json-globe-index --heatmap-dir /var/globe_history --write-state /var/globe_history --heatmap 5 --json-trace-interval 5"

pi@pi:/var/globe_history/2024/02/19/heatmap $ ls -l
total 32
-rw-r--r-- 1 readsb nogroup  977 Feb 19 11:58 20.bin.ttf
-rw-r--r-- 1 readsb nogroup  398 Feb 19 12:58 22.bin.ttf
-rw-r--r-- 1 readsb nogroup 8424 Feb 19 13:58 24.bin.ttf
-rw-r--r-- 1 readsb nogroup  396 Feb 19 19:28 35.bin.ttf
-rw-r--r-- 1 readsb nogroup 1916 Feb 19 19:50 36.bin.ttf
-rw-r--r-- 1 readsb nogroup 1845 Feb 19 23:39 44.bin.ttf
LoickFahl commented 7 months ago 
pi@pi:~ $ sudo ./gdb.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/bin/readsb --quiet --device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss --lat 000 --lon 000 --max-range 1000 --net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-            port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json-every 1 --range-outline-hours 168 --write-json-globe-index --heatmap-dir /var/globe_history --write-state /var/globe_history --heatmap 5 --json-trace-interval 5 --write-json /run/readsb
[2024-02-19 23:39:46.162 CET] readsb starting up.
readsb version: 3.14.1610 wiedehopf git: 11bf36c (committed: Sun Feb 18 18:37:48 2024 0100)
Using lat:  000, lon:    000
[New Thread 0xf7bb9440 (LWP 5705)]
[New Thread 0xf73b8440 (LWP 5706)]
[New Thread 0xf6bb7440 (LWP 5707)]
[New Thread 0xf63b6440 (LWP 5708)]
loading state .....
[New Thread 0xf58a9440 (LWP 5709)]
[New Thread 0xf50a8440 (LWP 5710)]
[New Thread 0xf48a7440 (LWP 5711)]
[New Thread 0xf40a6440 (LWP 5712)]
actual range outline, read bytes: 368644
[Thread 0xf50a8440 (LWP 5710) exited]
[Thread 0xf58a9440 (LWP 5709) exited]
[Thread 0xf48a7440 (LWP 5711) exited]
[Thread 0xf40a6440 (LWP 5712) exited]
 .......... done, loaded 308 aircraft in 0.050 seconds!
aircraft table fill: 0.0
[New Thread 0xf40a6440 (LWP 5713)]
[New Thread 0xf48a7440 (LWP 5714)]
[New Thread 0xf58a9440 (LWP 5716)]
[New Thread 0xf50a8440 (LWP 5715)]
[New Thread 0xf30ff440 (LWP 5717)]
30002: Raw TCP output port
30005: Beast TCP output port
30003: SBS TCP output ALL port
30001: Raw TCP input port
30004: Beast TCP input port
30104: Beast TCP input port
[New Thread 0xf28fe440 (LWP 5718)]
[New Thread 0xf20fd440 (LWP 5719)]
[New Thread 0xf16fc440 (LWP 5720)]
[New Thread 0xf0cff440 (LWP 5721)]
[Thread 0xf50a8440 (LWP 5715) exited]
[New Thread 0xf50a8440 (LWP 5722)]
[New Thread 0xf02ff440 (LWP 5723)]
[New Thread 0xefafe440 (LWP 5724)]
[New Thread 0xef2fd440 (LWP 5725)]
[Thread 0xf50a8440 (LWP 5722) exited]
[New Thread 0xf50a8440 (LWP 5726)]
[Thread 0xf50a8440 (LWP 5726) exited]
[New Thread 0xf50a8440 (LWP 5727)]
[Thread 0xf50a8440 (LWP 5727) exited]
[New Thread 0xf50a8440 (LWP 5728)]
rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
[New Thread 0xf50a8440 (LWP 5734)]
[Thread 0xf50a8440 (LWP 5728) exited]
Found Rafael Micro R820T tuner
rtlsdr: tuner gain set to 25.4 dB
Allocating 16 zero-copy buffers
Wrote live traces for aircraft active within the last 15 minutes. This took 4.0 seconds (roughly 0 minutes).
Wrote live traces for aircraft active within the last 24 hours. This took 4.0 seconds (roughly 0 minutes).

Thread 12 "readsb" received signal SIGBUS, Bus error.
[Switching to Thread 0xf58a9440 (LWP 5716)]
0x0005add8 in handleHeatmap (now=<optimized out>) at globe_index.c:3239
3239    globe_index.c: No such file or directory.
#0  0x0005add8 in handleHeatmap (now=<optimized out>) at globe_index.c:3239
        state = 0xf0d07f0f
        all = 0xf0d07f8f
        addrtype_5bits = <optimized out>
        i = 4
        tb = {len = 61584, trace = 0x1a}
        squawk = 0
        callsign_interval = 0
        next_callsign = 1708381800000
        next = 1708383105000
        slice = 4000378888
        callsign = 146028888461
        a = 0xf58a50f0
        j = <optimized out>
        nowish = 1708381800
        utc = {tm_sec = 0, tm_min = 30, tm_hour = 22, tm_mday = 19, tm_mon = 1, tm_year = 124, tm_wday = 1, tm_yday = 49, tm_                                                                                                                                                                                                isdst = 0, tm_gmtoff = 0, tm_zone = 0xf7d5ea00 "GMT"}
        half_hour = <optimized out>
        start = 1708383600000
        end = -1094235758669070336
        num_slices = <optimized out>
        pathbuf = "e/blob_55.zstl\000\000/run/readsb/outline.json\000bin.ttf\000\377\377\377", '\000' <repeats 40 times>, "\2                                                                                                                                                                                                00w\327\367\000\000\000\000`\234\327\367", '\000' <repeats 16 times>, "actual range outline, read bytes: 368644\n", '\000' <r                                                                                                                                                                                                epeats 215 times>...
        tmppath = "e/blob_55.zstl.readsb_tmp\000^\000outl/var/globe_history/internal_state/rangeDirs.gz\000B[\372\334@p\224\0                                                                                                                                                                                                00\000pM\332G\204\263,B$\337\327@\240\214\000\000MP\003H\374M,BS\314\337@\210\220\000\000\376\242\365G9b,Bnt\334@\210\220\000                                                                                                                                                                                                \000Y\004\337G\034},B\322\376\327@\210\220\000\000\247-\323G\354\201,B\t}\325@\240\214\000\000\347n\321G\234t,B\304\331\324@\                                                                                                                                                                                                240\214\000\000\346\354\026H\240\233+B0\017\345@\210\220\000\000"...
        len = 5
        len2 = 0
        alloc = -1256433892932452091
        buffer = 0xf37fd0e8
        slices = 0x0
        passbuffer = {buf = 0xf0d07e60, size = 5906, cctx = 0x0, dctx = 0x0}
        buffer2 = <optimized out>
        indexSize = <optimized out>
        index = <optimized out>
        base_dir = <optimized out>
        dateDir = "\024\243\320\360\004\031\000\000\000\000\000\000\070\001\000\000\000c\217\345\020\020\217\354P\220\320\360                                                                                                                                                                                                \034j\320\360Hh\320\360\034j\320\360Hh\320\360\330h\320\360\000\000\000\000T\224\320\360\204\207\343\367\031\020\217\354\367\                                                                                                                                                                                                277\300\000.\276\320\360\b\000\000\000\024\243\320\360\004\031", '\000' <repeats 22 times>, "\a\360g\355\031\020\217\354,\276                                                                                                                                                                                                \320\360\030\276\320\360(\276\320\360*\276\320\360#\020\217\354\n\000\000\000\064\242\320\360T\224\320\360\260g\320\360\b\000                                                                                                                                                                                                \000\000`R\212\365\224j\320\360\020\360g\355\000\000\000\000\000\000\000\000\004\000\000\000\b\360g\355,\000\000\000\000c\217                                                                                                                                                                                                \345\t\000\000\000\034j\320\360Hh\320\360"...
        fd = <optimized out>
#1  0x00018cfc in miscStuff (now=<optimized out>) at readsb.c:2338
        next_clients_json = 1708383607762
#2  0x000190f8 in miscEntryPoint (arg=<optimized out>) at readsb.c:2389
        watch = {tv_sec = 13590, tv_nsec = 981872113}
        start_time = {tv_sec = 5, tv_nsec = 138922812}
        antiSpam2 = 0
        now = 1708383600162
        elapsed = <optimized out>
        ts = {tv_sec = 1708383600, tv_nsec = 162814496}
#3  0xf7f66310 in start_thread (arg=0xf58a9440) at pthread_create.c:477
        ret = <optimized out>
        pd = 0xf58a9440
        unwind_buf = {cancel_jmp_buf = {{jmp_buf = {59480389, 32994777, -175467456, -73484, -175467456, 338, -73486, -73484,                                                                                                                                                                                                 -175468672, -175468676, 0, 0, 262796, 0 <repeats 51 times>}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data =                                                                                                                                                                                                 {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
        not_first_call = <optimized out>
#4  0xf7d05598 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
No locals.
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

after the crash new Json

JSON_OPTIONS="--json-location-accuracy 2 --write-json-every 1 --range-outline-hours 72 --write-json=/run/readsb/ --heatmap-dir /var/globe_history --heatmap 5 --json-trace-interval 5"

we'll see if it hold during the night

pi@pi:/var/globe_history/2024/02 $ find /var/globe_history | grep heat
/var/globe_history/2024/02/19/heatmap
/var/globe_history/2024/02/19/heatmap/24.bin.ttf
/var/globe_history/2024/02/19/heatmap/35.bin.ttf
/var/globe_history/2024/02/19/heatmap/20.bin.ttf
/var/globe_history/2024/02/19/heatmap/22.bin.ttf
/var/globe_history/2024/02/19/heatmap/36.bin.ttf
/var/globe_history/2024/02/19/heatmap/44.bin.ttf
/var/globe_history/2024/02/19/heatmap/45.bin.ttf

pi@pi:/var/globe_history/2024/02/19/heatmap $ ls -l
total 36
-rw-r--r-- 1 readsb nogroup  977 Feb 19 11:58 20.bin.ttf
-rw-r--r-- 1 readsb nogroup  398 Feb 19 12:58 22.bin.ttf
-rw-r--r-- 1 readsb nogroup 8424 Feb 19 13:58 24.bin.ttf
-rw-r--r-- 1 readsb nogroup  396 Feb 19 19:28 35.bin.ttf
-rw-r--r-- 1 readsb nogroup 1916 Feb 19 19:50 36.bin.ttf
-rw-r--r-- 1 readsb nogroup 1845 Feb 19 23:39 44.bin.ttf
-rw-r--r-- 1 readsb nogroup 1059 Feb 20 00:25 45.bin.ttf
wiedehopf commented 7 months ago

To be honest this doesn't really help me debug the error ... but it's a start i guess.

The container version run by many people defaults to heatmap enabled so the codepath should be somewhat well used. Not saying the code is good btw :)

I pushed a new version kinda blindly modifying some checks that might get me closer to the actual problem. Can you get the autoinstall script but change the branch? https://github.com/wiedehopf/adsb-scripts/raw/master/readsb-install.sh ---> but change BRANCH=stale to BRANCH=sid Then run it to overwrite your currently installed binary.

I'm tempted to just say your hardware is broken somehow ... but yeah no clue. Some of the variables from the backtrace are completely nonsensical.

LoickFahl commented 7 months ago

Well, let's see, Branch changed 

}
BRANCH="sid"
if grep -E 'wheezy|jessie' /etc/os-release -qs; then
    BRANCH="jessie"
fi
if ! getGIT "$repository" "$BRANCH" "$ipath/git"
then
    echo "Unable to git clone the repository"
    exit 1
fi

Let's run it

pi@pi:~ $ sudo nano readsb-install.sh
pi@pi:~ $ sudo chmod 777 readsb-install.sh
pi@pi:~ $ sudo ./readsb-install.sh
23976 (process ID) old priority 0, new priority 10
remote: Enumerating objects: 94, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (46/46), done.
remote: Total 53 (delta 41), reused 11 (delta 4), pack-reused 0
Unpacking objects: 100% (53/53), 32.45 KiB | 84.00 KiB/s, done.
From https://github.com/wiedehopf/readsb
 * branch            sid        -> FETCH_HEAD
HEAD is now at d275321 debugging heatmap
rm -f *.o uat2esnt/*.o compat/clock_gettime/*.o compat/clock_nanosleep/*.o readsb viewadsb cprtests crctests convert_benchmark
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c argp.c -o argp.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c anet.c -o anet.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c interactive.c -o interactive.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c mode_ac.c -o mode_ac.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c mode_s.c -o mode_s.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c comm_b.c -o comm_b.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c json_out.c -o json_out.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (commit              ted: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-tr              uncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c net_io.c -o net_io.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c crc.c -o crc.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c demod_2400.c -o demod_2400.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c uat2esnt/uat2esnt.c -o uat2esnt/uat2esnt.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c uat2esnt/uat_decode.c -o uat2esnt/uat_decode.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c stats.c -o stats.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c cpr.c -o cpr.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c icao_filter.c -o icao_filter.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c track.c -o track.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c util.c -o util.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c fasthash.c -o fasthash.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c convert.c -o convert.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c sdr_ifile.c -o sdr_ifile.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c sdr_beast.c -o sdr_beast.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c sdr.c -o sdr.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c ais_charset.c -o ais_charset.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c globe_index.c -o globe_index.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c geomag.c -o geomag.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 26 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c receiver.c -o receiver.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c aircraft.c -o aircraft.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c api.c -o api.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c minilzo/minilzo.c -o minilzo.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c threadpool.c -o threadpool.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c sdr_rtlsdr.c -o sdr_rtlsdr.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)"\" -Wdate-time -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DAIRCRAFT_HASH_BITS=16 -DENABLE_RTLSDR  -g -O2 -march=native -mno-unaligned-access -c readsb.c -o readsb.o
cc -o readsb readsb.o argp.o anet.o interactive.o mode_ac.o mode_s.o comm_b.o json_out.o net_io.o crc.o demod_2400.o uat2esnt/uat2esnt.o uat2esnt/uat_decode.o stats.o cpr.o icao_filter.o track.o util.o fasthash.o convert.o sdr_ifile.o sdr_beast.o sdr.o ais_charset.o globe_index.o geomag.o receiver.o aircraft.o api.o minilzo.o threadpool.o sdr_rtlsdr.o  -pthread -lpthread -lm -lrt -lzstd -lz -lncurses -lrtlsdr -lusb-1.0 -O2 -march=native -mno-unaligned-access
rm -f viewadsb
cp readsb viewadsb
--------------
--2024-02-20 01:28:24--  https://raw.githubusercontent.com/wiedehopf/tar1090/master/install.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8003::154, 2606:50c0:8000::154, 2606:50c0:8001::154, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8003::154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18363 (18K) [text/plain]
Saving to: ‘tar1090-install.sh’

tar1090-install.sh                      100%[============================================================================>]  17.93K  --.-KB/s    in 0.02s

2024-02-20 01:28:25 (878 KB/s) - ‘tar1090-install.sh’ saved [18363/18363]

27851 (process ID) old priority 10, new priority 10
HEAD is now at 9f81854 database update (to keep the repository small, this commit is replaced regularly)
HEAD is now at 6624e19 allow longer seen / seen_pos in binCraft version
Restarting lighttpd ...
--------------
--------------
All done! Webinterface available at http://192.168.1.200/tar1090
Don't forget to set your location using decimal latitude and longitude:

sudo readsb-set-location 50.12344 10.23429

Json update

JSON_OPTIONS="--json-location-accuracy 2 --write-json-every 1 --range-outline-hours 72 --write-json=/run/readsb/ --heatmap-dir /var/globe_history/ --write-state /var/globe_history --heatmap 5 --json-trace-interval 5"

New heatmap file, that's encouraging 

pi@pi:~ $ find /var/globe_history | grep heat
/var/globe_history/2024/02/19/heatmap
/var/globe_history/2024/02/19/heatmap/24.bin.ttf
/var/globe_history/2024/02/19/heatmap/35.bin.ttf
/var/globe_history/2024/02/19/heatmap/20.bin.ttf
/var/globe_history/2024/02/19/heatmap/22.bin.ttf
/var/globe_history/2024/02/19/heatmap/36.bin.ttf
/var/globe_history/2024/02/19/heatmap/44.bin.ttf
/var/globe_history/2024/02/19/heatmap/45.bin.ttf

And the script

pi@pi:~ $ sudo ./gdb.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/bin/readsb --quiet --device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss --lat aaa --lon aaaa --max-range 1000 --net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json-every 1 --range-outline-hours 72 --write-json=/run/readsb/ --heatmap-dir /var/globe_history/ --write-state /var/globe_history --heatmap 5 --json-trace-interval 5 --write-json /run/readsb
[2024-02-20 01:37:33.666 CET] readsb starting up.
readsb version: 3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20 01:08:30 2024 0100)
Using lat:   aaaa, lon:    aaaa
[New Thread 0xf7bb9440 (LWP 12968)]
loading state .....
[New Thread 0xf70ac440 (LWP 12969)]
[New Thread 0xf68ab440 (LWP 12970)]
[New Thread 0xf60aa440 (LWP 12971)]
[New Thread 0xf58a9440 (LWP 12972)]
actual range outline, read bytes: 368644
[Thread 0xf58a9440 (LWP 12972) exited]
[Thread 0xf68ab440 (LWP 12970) exited]
[Thread 0xf70ac440 (LWP 12969) exited]
[Thread 0xf60aa440 (LWP 12971) exited]
 .......... done, loaded 9 aircraft in 0.067 seconds!
aircraft table fill: 0.0
[New Thread 0xf58a9440 (LWP 12973)]
[New Thread 0xf60aa440 (LWP 12974)]
30002: Raw TCP output port
[New Thread 0xf68ab440 (LWP 12975)]
30005: Beast TCP output port
[New Thread 0xf70ac440 (LWP 12976)]
30003: SBS TCP output ALL port
30001: Raw TCP input port
30004: Beast TCP input port
30104: Beast TCP input port
[New Thread 0xf4abe440 (LWP 12977)]
[New Thread 0xf42bd440 (LWP 12978)]
[New Thread 0xf3abc440 (LWP 12979)]
[Thread 0xf70ac440 (LWP 12976) exited]
[New Thread 0xf70ac440 (LWP 12980)]
[New Thread 0xf70ac440 (LWP 12981)]
[Thread 0xf70ac440 (LWP 12980) exited]
[Thread 0xf70ac440 (LWP 12981) exited]
[New Thread 0xf70ac440 (LWP 12982)]
[Thread 0xf70ac440 (LWP 12982) exited]
[New Thread 0xf70ac440 (LWP 12983)]
[Thread 0xf70ac440 (LWP 12983) exited]
rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
[New Thread 0xf70ac440 (LWP 12987)]
Found Rafael Micro R820T tuner
rtlsdr: tuner gain set to 25.4 dB
Allocating 16 zero-copy buffers

Now let's see if it hold

LoickFahl commented 7 months ago

Here with good news. Looks like it's working , i will try later this week to have it save on an external drive to spare the W/R on the SD card.

Thank you for your help.

Current config

# readsb configuration
# This is sourced by /etc/systemd/system/default.target.wants/readsb.service as
# daemon startup configuration.
#Basestation
RECEIVER_OPTIONS="--device SDR000 --device-type rtlsdr --gain 25.4 --ppm 0 --mlat --forward-mlat --gnss"
DECODER_OPTIONS="--lat xxx --lon xxx --max-range 1000"
NET_OPTIONS="--net --net-heartbeat 60 --net-ro-size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005"
JSON_OPTIONS="--json-location-accuracy 2 --write-json-every 1 --range-outline-hours 72 --write-json-globe-index --write-json=/run/readsb/ --heatmap-dir /var/globe_history/ --write-state /var/globe_history --heatmap 5 --json-trace-interval 5"
pi@pi:/var/globe_history/2024/02/20 $ ls
heatmap
pi@pi:~ $ find /var/globe_history | grep heat
xxxxxxxxxxxx
/var/globe_history/2024/02/20/heatmap
/var/globe_history/2024/02/20/heatmap/27.bin.ttf
/var/globe_history/2024/02/20/heatmap/33.bin.ttf
/var/globe_history/2024/02/20/heatmap/24.bin.ttf
/var/globe_history/2024/02/20/heatmap/16.bin.ttf
/var/globe_history/2024/02/20/heatmap/10.bin.ttf
/var/globe_history/2024/02/20/heatmap/02.bin.ttf
/var/globe_history/2024/02/20/heatmap/08.bin.ttf
/var/globe_history/2024/02/20/heatmap/15.bin.ttf
/var/globe_history/2024/02/20/heatmap/26.bin.ttf
/var/globe_history/2024/02/20/heatmap/32.bin.ttf
/var/globe_history/2024/02/20/heatmap/04.bin.ttf
/var/globe_history/2024/02/20/heatmap/12.bin.ttf
/var/globe_history/2024/02/20/heatmap/20.bin.ttf
/var/globe_history/2024/02/20/heatmap/22.bin.ttf
/var/globe_history/2024/02/20/heatmap/25.bin.ttf
/var/globe_history/2024/02/20/heatmap/17.bin.ttf
/var/globe_history/2024/02/20/heatmap/30.bin.ttf
/var/globe_history/2024/02/20/heatmap/00.bin.ttf
/var/globe_history/2024/02/20/heatmap/21.bin.ttf
/var/globe_history/2024/02/20/heatmap/11.bin.ttf
/var/globe_history/2024/02/20/heatmap/31.bin.ttf
/var/globe_history/2024/02/20/heatmap/07.bin.ttf
/var/globe_history/2024/02/20/heatmap/06.bin.ttf
/var/globe_history/2024/02/20/heatmap/13.bin.ttf
/var/globe_history/2024/02/20/heatmap/28.bin.ttf
/var/globe_history/2024/02/20/heatmap/03.bin.ttf
/var/globe_history/2024/02/20/heatmap/19.bin.ttf
/var/globe_history/2024/02/20/heatmap/29.bin.ttf
/var/globe_history/2024/02/20/heatmap/14.bin.ttf
/var/globe_history/2024/02/20/heatmap/09.bin.ttf
/var/globe_history/2024/02/20/heatmap/05.bin.ttf
/var/globe_history/2024/02/20/heatmap/18.bin.ttf
/var/globe_history/2024/02/20/heatmap/23.bin.ttf
/var/globe_history/2024/02/20/heatmap/01.bin.ttf
wiedehopf commented 7 months ago

You can either use a mount point to have it in /var/globe_history or you can change the webserver configuration so serve the files from another directory. I suppose /var/globe_history itself being a symlink to somewhere else could also work, not sure about that.

Still can't really imagine how the debugging info makes any sense. Well we'll see i guess.

LoickFahl commented 7 months ago

Okay, now that i've solved my usb drives issues and reformatted the good one 

pi@pi:~ $ lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdd           8:48   1 58.6G  0 disk
└─sdd1        8:49   1 58.6G  0 part
mmcblk0     179:0    0 59.4G  0 disk
├─mmcblk0p1 179:1    0  2.8G  0 part
├─mmcblk0p2 179:2    0    1K  0 part
├─mmcblk0p5 179:5    0   32M  0 part
├─mmcblk0p6 179:6    0  256M  0 part /boot
└─mmcblk0p7 179:7    0 56.3G  0 part /
pi@pi:/mnt $ sudo chown -R readsb:root /mnt/globe_history
pi@pi:/mnt $ ls -l
total 8
drwxr-xr-x 2 root   root 4096 Feb 22 17:24 auto_rx_logs
drwxr-xr-x 4 readsb root 4096 Feb 22 17:16 globe_history
pi@pi:/mnt $ sudo mv /var/globe_history/* /mnt/globe_history/
pi@pi:/mnt/globe_history/2024/02/22/heatmap $ ls
31.bin.ttf  32.bin.ttf
pi@pi:~ $ sudo ./gdb.sh
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.s                                                                                                                                                                                                                                                     o.1".
invoked by: /usr/bin/readsb --quiet --device SDR000 --device-type rtlsdr                                                                                                                                                                                                                                                      --gain 22.9 --ppm 0 --lat xxxx --lon xxxx --max-range 1000 --mla                                                                                                                                                                                                                                                     t --forward-mlat --gnss --modeac-auto --net --net-heartbeat 60 --net-ro-                                                                                                                                                                                                                                                     size 1250 --net-ro-interval 0.05 --net-ri-port 30001 --net-ro-port 30002                                                                                                                                                                                                                                                      --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --js                                                                                                                                                                                                                                                     on-location-accuracy 2 --write-json-every 1 --range-outline-hours 72 --w                                                                                                                                                                                                                                                     rite-json-globe-index --write-json=/run/readsb/ --heatmap-dir /mnt/globe                                                                                                                                                                                                                                                     _history/ --write-state /mnt/globe_history --heatmap 10 --json-trace-int                                                                                                                                                                                                                                                     erval 10 --write-json /run/readsb
[2024-02-22 17:10:32.748 CET] readsb starting up.
readsb version: 3.14.1610 wiedehopf git: d275321 (committed: Tue Feb 20                                                                                                                                                                                                                                                      01:08:30 2024 0100)
Using lat:   xxxxx, lon:    xxxxx
[New Thread 0xf7bb9440 (LWP 31484)]
[New Thread 0xf73b8440 (LWP 31485)]
[New Thread 0xf6bb7440 (LWP 31486)]
[New Thread 0xf63b6440 (LWP 31487)]
loading state .....
[New Thread 0xf58a9440 (LWP 31489)]
[New Thread 0xf50a8440 (LWP 31490)]
[New Thread 0xf48a7440 (LWP 31491)]
[New Thread 0xf40a6440 (LWP 31492)]
actual range outline, read bytes: 368644
[Thread 0xf50a8440 (LWP 31490) exited]
[Thread 0xf58a9440 (LWP 31489) exited]
[Thread 0xf48a7440 (LWP 31491) exited]
[Thread 0xf40a6440 (LWP 31492) exited]
 .......... done, loaded 1146 aircraft in 0.298 seconds!
aircraft table fill: 0.0
[New Thread 0xf40a6440 (LWP 31513)]
[New Thread 0xf48a7440 (LWP 31514)]
[New Thread 0xf50a8440 (LWP 31515)]
30002: Raw TCP output port
30005: Beast TCP output port
30003: SBS TCP output ALL port
30001: Raw TCP input port
[New Thread 0xf58a9440 (LWP 31517)]
30004: Beast TCP input port
30104: Beast TCP input port
[New Thread 0xf26ff440 (LWP 31518)]
[New Thread 0xf1efe440 (LWP 31519)]
[New Thread 0xf16fd440 (LWP 31520)]
[New Thread 0xf0efc440 (LWP 31521)]
[New Thread 0xf02ff440 (LWP 31522)]
[Thread 0xf50a8440 (LWP 31515) exited]
[New Thread 0xf50a8440 (LWP 31523)]
[New Thread 0xefafe440 (LWP 31524)]
[New Thread 0xef2fd440 (LWP 31525)]
[New Thread 0xeeafc440 (LWP 31526)]
[Thread 0xf50a8440 (LWP 31523) exited]
[New Thread 0xf50a8440 (LWP 31527)]
[Thread 0xf50a8440 (LWP 31527) exited]
[New Thread 0xf50a8440 (LWP 31528)]
[Thread 0xf50a8440 (LWP 31528) exited]
[New Thread 0xf50a8440 (LWP 31535)]
[Thread 0xf50a8440 (LWP 31535) exited]
rtlsdr: using device #0: Generic RTL2832U OEM (Realtek, RTL2838UHIDIR, SN SDR000)
[New Thread 0xf50a8440 (LWP 31536)]
Found Rafael Micro R820T tuner
rtlsdr: tuner gain set to 22.9 dB
Allocating 16 zero-copy buffers
Wrote live traces for aircraft active within the last 15 minutes. This took 4.0 seconds (roughly 0 minutes).
Wrote live traces for aircraft active within the last 24 hours. This took 7.1 seconds (roughly 0 minutes).
FIFO dropped, suppressing this message for 30 seconds.
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/1/1470 ms                                                                                                                                                                                                                                                     , suppressing for 5 seconds!
wiedehopf commented 7 months ago

Regarding the files being served by the webserver, see my previous post. Pretty sure you have done nothing to have it be served.

As i mentioned you can try a symlink:

sudo rm -rf /var/globe_history
sudo ln -s /mnt/globe_history /var/globe_history
LoickFahl commented 7 months ago

The heatmap files are being created without issues it seems

pi@pi:/mnt/globe_history/2024/02/22/heatmap $ ls -l
total 200
-rw-r--r-- 1 readsb nogroup  1845 Feb 22 17:25 31.bin.ttf
-rw-r--r-- 1 readsb nogroup 35324 Feb 22 17:41 32.bin.ttf
-rw-r--r-- 1 readsb nogroup 41763 Feb 22 18:13 33.bin.ttf
-rw-r--r-- 1 readsb nogroup 34193 Feb 22 18:45 34.bin.ttf
-rw-r--r-- 1 readsb nogroup 41042 Feb 22 19:02 35.bin.ttf
-rw-r--r-- 1 readsb nogroup 32923 Feb 22 19:42 36.bin.ttf
LoickFahl commented 7 months ago

Some new info, everything seems to work fine but i've lost the MLAT. I'm not yet sure if it's related to the this issue or not.

wiedehopf commented 7 months ago

To be honest i wouldn't be surprised if 

sudo dmesg | grep voltage

showed undervoltage.

Or your hardware is problematic in some other way .... Oh right USB stick, if the writes are in bursts that will cause packet loss for the SDR, really you can't really use USB for storage while using an SDR.