search

wiedehopf / readsb

ADS-B decoder swiss knife
Other
360 stars 79 forks source link

Crashed with error message "High load: heatmap_and_stuff took 4113 ms!" #6

Closed bive242 closed 3 years ago

bive242 commented 3 years ago

I'm running dev-branch and that may explain why there are errors, but I'll report it anyway.

Readsb crashed and this is from journalctl:

Sep 24 16:32:52 Spy readsb[26751]: 30104: Beast TCP input port
Sep 24 21:23:03 Spy readsb[26751]: High load: heatmap_and_stuff took 4113 ms! Suppressing for 30 seconds
Sep 24 21:23:26 Spy readsb[26751]: FATAL: trackPeriodicUpdate() interval 11.1 seconds! Trying for an orderly shutdown as well as possible!
Sep 24 21:23:26 Spy readsb[26751]: lockThreads() probably hung on statsWrite
Sep 24 21:23:28 Spy readsb[26751]: upkeep thread: threadSignalJoin timed out after 2.0 seconds, undefined behaviour may result!
Sep 24 21:23:28 Spy readsb[26751]: saving state .....
Sep 24 21:25:30 Spy readsb[26751]:  .......... done, saved 52 aircraft in 121.898 seconds!
Sep 24 21:25:30 Spy readsb[26751]: FATAL: thread upkeep could not be joined, calling abort()!
Sep 24 21:25:30 Spy systemd[1]: readsb.service: Main process exited, code=killed, status=6/ABRT
Sep 24 21:25:30 Spy systemd[1]: readsb.service: Failed with result 'signal'.
Sep 24 21:26:00 Spy systemd[1]: readsb.service: Service RestartSec=30s expired, scheduling restart.
Sep 24 21:26:00 Spy systemd[1]: readsb.service: Scheduled restart job, restart counter is at 1.
Sep 24 21:26:00 Spy systemd[1]: Stopped readsb ADS-B receiver.
Sep 24 21:26:00 Spy systemd[1]: Started readsb ADS-B receiver.
Sep 24 21:26:00 Spy readsb[10692]: readsb version: wiedehopf git: f06374e (commited: Tue Sep 21 13:34:41 2021 0200)
Sep 24 21:26:00 Spy readsb[10692]: Using lat:   60.9211, lon:   16.6788
Sep 24 21:26:00 Spy readsb[10692]: loading state .....
Sep 24 21:26:00 Spy readsb[10692]:  .......... done, loaded 52 aircraft in 0.006 seconds!
Sep 24 21:26:00 Spy readsb[10692]: aircraft table fill: 0.0
Sep 24 21:26:00 Spy readsb[10692]: actual range outline, read bytes: 144004
Sep 24 21:26:00 Spy readsb[10692]: Database update done!
Sep 24 21:26:00 Spy readsb[10692]: 30002: Raw TCP output port            30005: Beast TCP output port
Sep 24 21:26:00 Spy readsb[10692]: 30003: SBS TCP output ALL port        30004: Beast TCP input port
Sep 24 21:26:00 Spy readsb[10692]: 30104: Beast TCP input port

I'm running with the following arguments:

/usr/bin/readsb --net-only --write-json-every 1 --db-file tar1090 --lat XX.XXXXX --lon YY.YYYYY --max-range 720 --heatmap-dir /var/globe_history --heatmap 30 --write-state=/var/globe_history --write-prom=/usr/share/dump1090-fa/html/data-prom --net --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.05 --net-ri-port 0 --net-ro-port 30002 --net-sbs-port 30003 --net-bi-port 30004,30104 --net-bo-port 30005 --json-location-accuracy 2 --write-json /run/readsb --quiet

Something happened that consumed a lot of CPU (the system has approx 50% CPU usage from airspy_adsb, no other jobs are running besides generating graphs for graphs1090 and some statistical scripts, but they run only once per day and at another time.

wiedehopf commented 3 years ago

I'd say the write of the prom file hung: --write-prom=/usr/share/dump1090-fa/html/data-prom That's part of statsWrite.

Write stast / promfile to /run/readsb/stats.prom instead. Then you can get it at /tar1090/data/stats.prom if you have set tar1090 up to use readsb data.

It actually performed as expected .... The state was written to disk .. so i wouldn't even expect there to be any data loss.

Really if it hangs for 10 seconds writing to disk .... don't make readsb write to disk.

bive242 commented 3 years ago

I figured that with my low traffic count there really shouldn't be any problems writing directly to disk. Apparently there was, despite using a SSD disk on a sata600 controller.

It didn't work out the way I wanted it anyway, so I'll try another way :)

wiedehopf commented 3 years ago

I suppose there shouldn't be. But i don't see any issues with the code either and it's running basically the same version on a few servers.

bive242 commented 3 years ago

Well, I'm not digging any deeper into this. Just experimenting with some of the features readsb offers to see if they can be useful for my little project. I found some interesting scripts that I wanted to try out, but they didn't do what I want to do, so I'll do some quick and dirty parsing of aircraft.json and extract the relevant data from that and dump in a database. Then I'll think about a presentation layer.

wiedehopf commented 3 years ago

To write traces you're missing a couple options

wiedehopf commented 3 years ago

See this: https://github.com/wiedehopf/tar1090#0800-destroy-sd-card