sdr-enthusiasts / docker-readsb-protobuf

Multi-architecture readsb-protobuf container with support for RTLSDR, bladeRF and plutoSDR (x86_64, arm32v7, arm64v8)
231 stars 37 forks source link

ADSBexchange container unhealthy, beast connection #198

Open steinauf85 opened 1 year ago

steinauf85 commented 1 year ago

I've been running a stack of containers within Portainer for about a month now. Recently, I noticed that the adsbexchange keeps saying it;s unhealthy, but my other containers in the stack show as fine (readsb, fr24, piaware). Is there something I should do about this? Checking https://www.adsbexchange.com/myip/ says my feed and stats are ok.

Here is the error message for the container. Portainer reports container is Unhealthy with 14 failures in the past two hours.

adsbexchange-feed last updated: 1680290928.777, now: 1680290929.425652514, delta: .648652514. HEALTHY 
ADSBx reports beast connection: UNHEALTHY
ADSBx reports MLAT connection: HEALTHY 
listening for beast connections on port 30005. HEALTHY 
TCP4 listening on ANY:30005 (tcp): PASS 
listening for mlat connections on port 30105. HEALTHY 
TCP4 listening on ANY:30105 (tcp): PASS 
adsbexchange-feed error deaths: 0. HEALTHY 
adsbexchange-stats error deaths: 0. HEALTHY 
healthcheck error deaths: 0. HEALTHY 
mlat-client error deaths: 0. HEALTHY

logs:

Traceback (most recent call last):
  File "/scripts/health_from_adsbexchange.py", line 9, in <module>
    stats_json = json.loads(stats_raw.decode('utf-8'))
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[adsbexchange-feed] Statistics: Fri Mar 31 15:19:15 2023 EDT - Fri Mar 31 15:34:15 2023 EDT
[adsbexchange-feed] Network:
[adsbexchange-feed]   0.424099 MBytes received
[adsbexchange-feed]   0.194607 MBytes sent
[adsbexchange-feed] Messages from network clients:
[adsbexchange-feed]   0 Mode A/C messages received
[adsbexchange-feed]   0 Mode A/C messages received
[adsbexchange-feed]   22903 Mode S messages received
[adsbexchange-feed]     0 with bad message format or invalid CRC
[adsbexchange-feed]     12 with unrecognized ICAO address
[adsbexchange-feed]     22891 accepted with correct CRC
[adsbexchange-feed]     0 accepted with 1-bit error repaired
[adsbexchange-feed] 22903 total usable messages
[adsbexchange-feed] 0 surface position messages received
[adsbexchange-feed] 2395 airborne position messages received
[adsbexchange-feed] 2167 global CPR attempts with valid positions
[adsbexchange-feed] 0 global CPR attempts with bad data
[adsbexchange-feed]   0 global CPR attempts that failed the range check
[adsbexchange-feed]   0 global CPR attempts that failed the speed check
[adsbexchange-feed] 1 global CPR attempts with insufficient data
[adsbexchange-feed] 173 local CPR attempts with valid positions
[adsbexchange-feed]   173 aircraft-relative positions
[adsbexchange-feed]   0 receiver-relative positions
[adsbexchange-feed] 55 local CPR attempts that did not produce useful positions
[adsbexchange-feed]   0 local CPR attempts that failed the range check
[adsbexchange-feed]   0 local CPR attempts that failed the speed check
[adsbexchange-feed] 0 CPR messages that look like transponder failures filtered
[adsbexchange-feed] 0 non-ES altitude messages from ES-equipped aircraft ignored
[adsbexchange-feed] 19 unique aircraft tracks
[adsbexchange-feed] 0 aircraft tracks where only one message was seen
[adsbexchange-feed] CPU load: 0.3%
[adsbexchange-feed]   0 ms for demodulation
[adsbexchange-feed]   0 ms for reading from USB
[adsbexchange-feed]   2688 ms for network input and background tasks
[mlat-client] Receiver: connected    25.3 msg/s received        7.9 msg/s processed (31%)
[mlat-client] Server:   ready         0.0 kB/s from server      0.1 kB/s to server
[mlat-client] Results:  0.0 positions/minute
[mlat-client] Aircraft: 1 of 5 Mode S, 4 of 8 ADS-B used
dziban303 commented 1 year ago

Without my having done any investigation of the python error, it may be related to some things having been changed on ADSBexchange's website upon which the healthcheck script relied. Others are having the same issue. It's been reported up the chain on the sdr-enthusiasts discord server.

kad commented 1 year ago

might be related, but I also see frequent crashes of the readsb which automatically restarts silently. this instability can lead to unhealthy reporting for dependant containers.

Apr 05 09:19:02 pi3 systemd-coredump[7554]: [🡕] Process 7541 (readsb) of user 0 dumped core.

                                           Stack trace of thread 23741:
                                           #0  0x00000000b6d493b0 n/a (/lib/arm-linux-gnueabihf/libc-2.31.so + 0x5c3b0)
                                           ELF object binary architecture: ARM