search

wiedehopf / readsb

ADS-B decoder swiss knife
Other
346 stars 78 forks source link

--write-json causes bus error #67

Closed vip9 closed 4 months ago

vip9 commented 5 months ago

I've decided to upgrade some packages on my rpi runniong fr24 feed, etc, and after a day, there was a problem with readsb service. After running from hand, the /usr/bin/readsb --device-type rtlsdr --write-json /tmp/a causes program to stop (SIGBUS), while /usr/bin/readsb --device-type rtlsdr seems work fine (but not generating files for tar).

gdb/strace says not much

`[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1". invoked by: /usr/bin/readsb --device-type rtlsdr --quiet --write-json /run/readsb [2024-04-23 17:50:21.421 CEST] readsb starting up. readsb version: 3.14.1621 wiedehopf git: e79d24a (committed: Sat Apr 20 00:06:35 2024 0200) [New Thread 0xf7be3440 (LWP 3557)] [New Thread 0xf70d6440 (LWP 3558)] [New Thread 0xf66ff440 (LWP 3559)] [New Thread 0xf5efe440 (LWP 3560)] [New Thread 0xf54ff440 (LWP 3561)] [New Thread 0xf4cbd440 (LWP 3562)] [New Thread 0xf44bc440 (LWP 3563)] [New Thread 0xf38ff440 (LWP 3564)]

Thread 8 "readsb" received signal SIGBUS, Bus error. [Switching to Thread 0xf44bc440 (LWP 3563)] 0xf7e5a30c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1

0 0xf7e5a30c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1

No symbol table info available.

1 0x00000000 in ?? ()

No symbol table info available. Backtrace stopped: previous frame identical to this frame (corrupt stack?)`

Tried upgrading from auto scripts, but nothing changed.

getState/ and aircraft.json are created.

Any hints how to debug it (apart of full clean image, reinstall)?

wiedehopf commented 5 months ago

You could try removing / reinstalling libzstd. (libarary binary corrupted possibly) It would fit with being able to create the other files but not the compressed ones.

wiedehopf commented 5 months ago

Also run with --quiet so you don't get as much output spam.

gdb --directory=/usr/local/share/adsb-wiki/readsb-install/git/

Could try to run gdb with that option to better show line numbers ... but not sure that will help any with the corrupted stack.

vip9 commented 5 months ago

After a fresh RPi image everything works. It was probably some libraries incompatibility.

stefanpuch commented 4 months ago

I'm facing exact the same problem but so far I don't want to reinstall my RPi from scratch. Using the adsbexchange installation scrips a got a git clone of version 3.14.1617

root@raspberrypi:/usr/local/share/adsbexchange/readsb-git# gdb --directory /usr/local/share/adsbexchange/readsb-git -q --args readsb  --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail
Reading symbols from readsb...done.
(gdb) run
Starting program: /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail
[2024-05-09 18:53:05.997 CEST] readsb starting up.
readsb version: 3.14.1617 adsbexchange git: b43adca (committed: Fri Mar 15 12:14:02 2024 -0700)
Using lat:   53.1524, lon:    8.1890
[New Thread 0xf7c4f440 (LWP 11869)]
[New Thread 0xf744e440 (LWP 11870)]
[New Thread 0xf6c4d440 (LWP 11871)]
30154: Beast TCP input port
[New Thread 0xf58fe440 (LWP 11872)]
[New Thread 0xf60ff440 (LWP 11874)]
[New Thread 0xf50fd440 (LWP 11873)]
[New Thread 0xf3cff440 (LWP 11876)]
[Thread 0xf50fd440 (LWP 11873) exited]
[New Thread 0xf46ff440 (LWP 11875)]
Beast TCP input: Connection established: 127.0.0.1 port 30005
[Thread 0xf46ff440 (LWP 11875) exited]
[New Thread 0xf32ff440 (LWP 11877)]

Thread 8 "readsb" received signal SIGBUS, Bus error.
[Switching to Thread 0xf3cff440 (LWP 11876)]
0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) bt
#0  0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

I'm not a gdb expert, but is there anything else I could try to track the problem down?

cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
uname -a
Linux raspberrypi 5.10.103-v8+ #1529 SMP PREEMPT Tue Mar 8 12:26:46 GMT 2022 aarch64 GNU/Linux
wiedehopf commented 4 months ago

I suppose you could try compiling https://github.com/facebook/zstd and installing that. Then recompile readsb.

I suppose you could compile with libasan and see if that gives any more info. For that, run this script with the argument "sanitize" https://raw.githubusercontent.com/wiedehopf/adsb-scripts/master/readsb-install.sh

Or just see what the script does and compile yourself.

Really armhf is annoying to work with cause valgrind doesn't even work. Maybe libasan will do the trick. Also i've recently tested on armhf but on later versions of Raspbian. So possibly that libzstd on armhf buster is just broken? I don't know.

stefanpuch commented 4 months ago

Short (intermediate) update:

root@raspberrypi:/usr/local/share/adsbexchange# ldd readsb-git/readsb
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf77f6000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xf77b6000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xf7734000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xf771d000)
        libzstd.so.1 => /usr/lib/arm-linux-gnueabihf/libzstd.so.1 (0xf7684000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xf765c000)
        libncurses.so.6 => /lib/arm-linux-gnueabihf/libncurses.so.6 (0xf762e000)
        libtinfo.so.6 => /lib/arm-linux-gnueabihf/libtinfo.so.6 (0xf75fd000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xf75ea000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xf75bd000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xf7470000)
        /lib/ld-linux-armhf.so.3 (0xf780b000)
root@raspberrypi:/usr/local/share/adsbexchange# strings readsb-git/readsb | grep libasan
libasan.so
root@raspberrypi:/usr/local/share/adsbexchange# gdb -q --args /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail
Reading symbols from /usr/local/share/adsbexchange/readsb-git/readsb...done.
(gdb) run
Starting program: /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail
[2024-05-10 10:17:30.791 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: 0d9bf67 (committed: Tue Apr 30 14:49:28 2024 0200)
Using lat: 53.15236, lon: 8.18903
[New Thread 0xf46ff400 (LWP 22228)]
[New Thread 0xf34ff400 (LWP 22229)]
[New Thread 0xf2cfe400 (LWP 22230)]
30154: Beast TCP input port
[New Thread 0xf1eff400 (LWP 22231)]
[New Thread 0xf14ff400 (LWP 22232)]
[New Thread 0xf0cfe400 (LWP 22233)]
[New Thread 0xf04fd400 (LWP 22234)]
[New Thread 0xeeeff400 (LWP 22236)]
[Thread 0xf04fd400 (LWP 22234) exited]
[New Thread 0xef8ff400 (LWP 22235)]
Beast TCP input: Connection established: 127.0.0.1 port 30005
[Thread 0xef8ff400 (LWP 22235) exited]

Thread 7 "readsb" received signal SIGBUS, Bus error.
[Switching to Thread 0xf0cfe400 (LWP 22233)]
0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) bt
#0  0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

$ dpkg -L libzstd1
/.
/usr
/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/libzstd.so.1.3.8
/usr/share
/usr/share/doc
/usr/share/doc/libzstd1
/usr/share/doc/libzstd1/changelog.Debian.gz
/usr/share/doc/libzstd1/changelog.gz
/usr/share/doc/libzstd1/copyright
/usr/lib/arm-linux-gnueabihf/libzstd.so.1
$ sudo dpkg -L libzstd-dev
/.
/usr
/usr/include
/usr/include/zbuff.h
/usr/include/zdict.h
/usr/include/zstd.h
/usr/include/zstd_errors.h
/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/libzstd.a
/usr/lib/arm-linux-gnueabihf/pkgconfig
/usr/lib/arm-linux-gnueabihf/pkgconfig/libzstd.pc
/usr/share
/usr/share/doc
/usr/share/doc/libzstd-dev
/usr/share/doc/libzstd-dev/README.Debian
/usr/share/doc/libzstd-dev/changelog.Debian.gz
/usr/share/doc/libzstd-dev/changelog.gz
/usr/share/doc/libzstd-dev/copyright
/usr/share/doc/libzstd-dev/examples
/usr/share/doc/libzstd-dev/examples/Makefile
/usr/share/doc/libzstd-dev/examples/README.md
/usr/share/doc/libzstd-dev/examples/dictionary_compression.c
/usr/share/doc/libzstd-dev/examples/dictionary_decompression.c
/usr/share/doc/libzstd-dev/examples/multiple_simple_compression.c
/usr/share/doc/libzstd-dev/examples/multiple_streaming_compression.c.gz
/usr/share/doc/libzstd-dev/examples/simple_compression.c
/usr/share/doc/libzstd-dev/examples/simple_decompression.c
/usr/share/doc/libzstd-dev/examples/streaming_compression.c
/usr/share/doc/libzstd-dev/examples/streaming_decompression.c
/usr/share/doc/libzstd-dev/examples/streaming_memory_usage.c.gz
/usr/share/doc/libzstd-dev/examples/utils.h.gz
/usr/lib/arm-linux-gnueabihf/libzstd.so
$
wiedehopf commented 4 months ago

Instead of building a debian package, just build without making the package? Then use make install for a /usr/local install (at least that's what make install usually does).

Maybe i just download Raspbian buster and give it a test on a local system. libzstd shouldn't have changed in a long time ... https://metadata.ftp-master.debian.org/changelogs//main/libz/libzstd/libzstd_1.3.8+dfsg-3+deb10u2_changelog

So i suppose it could be that i'm passing it wrong stuff somehow. I guess i can add some debug option to readsb to show which parameters libzstd is getting called with.

Will get back to you. Using the newer libzstd really doesn't help i suppose, only tells us where the issue is, so don't worry about installing it.

wiedehopf commented 4 months ago

Ok .. rebuild from git and run with --devel=debugZstd It will print before calling the library so we at least know which arguments the library is called with. And which library call causes the error.

wiedehopf commented 4 months ago

Actually installed armhf buster and can't reproduce the issue.

Let's make sure you're compiling the correct code, do the following:

cd
git clone https://github.com/wiedehopf/readsb
cd readsb
make -j4
mkdir run
./readsb --net --devel=debugZstd --write-json run --quiet

Let's also get some info on the binary and the library ... and the CPU i guess:

grep Model /proc/cpuinfo
ldd readsb
md5sum /lib/arm-linux-gnueabihf/libzstd.so.1
stefanpuch commented 4 months ago

OK, thanks for your support.

root@raspberrypi:/usr/local/share/adsbexchange/readsb-git# gdb -q --args /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail --devel=debugZstd
Reading symbols from /usr/local/share/adsbexchange/readsb-git/readsb...done.
(gdb) run
Starting program: /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail --devel=debugZstd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/adsbexchange/readsb-git/readsb --net --net-only --write-json /run/adsbexchange-feed --net-beast-reduce-interval 0.5 --net-connector feed1.adsbexchange.com,30004,beast_reduce_out,feed2.adsbexchange.com,64004 --net-heartbeat 60 --net-ro-size 1280 --net-ro-interval 0.2 --net-ro-port 0 --net-sbs-port 0 --net-bi-port 30154 --net-bo-port 0 --net-ri-port 0 --write-json-every 1 --lat 53.15236 --lon 8.18903 --max-range 450 --json-location-accuracy 2 --range-outline-hours 24 --net-connector 127.0.0.1,30978,uat_in,silent_fail --net-connector 127.0.0.1,30005,beast_in,silent_fail --devel=debugZstd
[2024-05-10 14:30:36.875 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)
Using lat: 53.15236, lon: 8.18903
[New Thread 0xf46ff400 (LWP 24225)]
[New Thread 0xf34ff400 (LWP 24226)]
[New Thread 0xf2cfe400 (LWP 24227)]
30154: Beast TCP input port
[New Thread 0xf1eff400 (LWP 24228)]
[New Thread 0xf14ff400 (LWP 24229)]
[New Thread 0xf0cfe400 (LWP 24230)]
calling ZSTD_createCCtx()
ZSTD_createCCtx() returned 0xf0103c00
[New Thread 0xefcff400 (LWP 24231)]
calling ZSTD_compressCCtx() with cctx 0xf0103c00 pbuffer->size    196 src.buffer 0xeff00100 src.len    112 level 1 src.buffer[0] 0xa7 cctx_first_byte 0x00
[New Thread 0xef4fe400 (LWP 24232)]
[Thread 0xef4fe400 (LWP 24232) exited]
[New Thread 0xeecfd400 (LWP 24233)]
Beast TCP input: Connection established: 127.0.0.1 port 30005
[Thread 0xeecfd400 (LWP 24233) exited]

Thread 7 "readsb" received signal SIGBUS, Bus error.
[Switching to Thread 0xf0cfe400 (LWP 24230)]
0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) bt
#0  0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

I just saw your last comment will redo this evenning but should be correct since I did a git pull before

root@raspberrypi:/usr/local/share/adsbexchange/readsb-git# git status
On branch dev
Your branch is up to date with 'origin/dev'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        build.sh

nothing added to commit but untracked files present (use "git add" to track)
root@raspberrypi:/usr/local/share/adsbexchange/readsb-git# cat build.sh
#!/bin/bash
umask 022
set -e

make clean
THREADS=$(( $(grep -c ^processor /proc/cpuinfo) - 1 ))
THREADS=$(( THREADS > 0 ? THREADS : 1 ))
CFLAGS="-O2 -march=native"

# disable unaligned access for arm 32bit ...
if uname -m | grep -qs -e arm -e aarch64 && gcc -mno-unaligned-access hello.c -o /dev/null &>/dev/null; then
    CFLAGS+=" -mno-unaligned-access"
fi

CFLAGS+=" -fsanitize=address -static-libasan"
make "-j${THREADS}" AIRCRAFT_HASH_BITS=12 OPTIMIZE="$CFLAGS"
stefanpuch commented 4 months ago

grep Model /proc/cpuinfo Model : Raspberry Pi 3 Model B Rev 1.2

ldd readsb root@raspberrypi:/usr/local/share/adsbexchange/readsb-git# ldd readsb /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf7aee000) libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xf7aae000) libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xf7a2c000) librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xf7a15000) libzstd.so.1 => /usr/lib/arm-linux-gnueabihf/libzstd.so.1 (0xf797c000) libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xf7954000) libncurses.so.6 => /lib/arm-linux-gnueabihf/libncurses.so.6 (0xf7926000) libtinfo.so.6 => /lib/arm-linux-gnueabihf/libtinfo.so.6 (0xf78f5000) libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xf78e2000) libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xf78b5000) libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xf7768000) /lib/ld-linux-armhf.so.3 (0xf7b03000) root@raspberrypi:/usr/local/share/adsbexchange/readsb-git#

md5sum /lib/arm-linux-gnueabihf/libzstd.so.1 root@raspberrypi:/usr/local/share/adsbexchange/readsb-git# md5sum /usr/lib/arm-linux-gnueabihf/libzstd.so.1 a90d4757f6a9281ed240f72b58c78d0e /usr/lib/arm-linux-gnueabihf/libzstd.so.1

--> is located in /usr/lib/...

wiedehopf commented 4 months ago

Did some more google ... let's get some more info

uname -a
wiedehopf commented 4 months ago

/lib will also work, there is symlink stuff going on. Anyhow the md5sum matches with what i have.

Please do a fresh clone, the adsbexchange-git directory makes me unhappy. (i guess it's unwarranted with the git version shown on startup .... grasping at straws here) Plus don't use the build script i want to go minimal reproducer:

cd
git clone https://github.com/wiedehopf/readsb
cd readsb

make clean
make -j4
mkdir run
./readsb --net --devel=debugZstd --write-json run --quiet
wiedehopf commented 4 months ago

I'm testing on the same exact model as well right now ...

Model           : Raspberry Pi 3 Model B Rev 1.2

I assume in regards to software you've updated it ... otherwise the binary wouldn't have changed and this whole issue wouldn't have happened.

stefanpuch commented 4 months ago 

root@raspberrypi:/usr/local/share# git clone https://github.com/wiedehopf/readsb.git
Cloning into 'readsb'...
remote: Enumerating objects: 8058, done.
remote: Counting objects: 100% (785/785), done.
remote: Compressing objects: 100% (339/339), done.
remote: Total 8058 (delta 505), reused 678 (delta 414), pack-reused 7273
Receiving objects: 100% (8058/8058), 2.78 MiB | 2.40 MiB/s, done.
Resolving deltas: 100% (5934/5934), done.
root@raspberrypi:/usr/local/share# cd readsb/
root@raspberrypi:/usr/local/share/readsb# make clean
rm -f *.o uat2esnt/*.o compat/clock_gettime/*.o compat/clock_nanosleep/*.o readsb viewadsb cprtests crctests convert_benchmark
root@raspberrypi:/usr/local/share/readsb# make j4
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c argp.c -o argp.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c anet.c -o anet.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c interactive.c -o interactive.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c mode_ac.c -o mode_ac.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c mode_s.c -o mode_s.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c comm_b.c -o comm_b.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c json_out.c -o json_out.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c net_io.c -o net_io.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c crc.c -o crc.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c demod_2400.c -o demod_2400.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c uat2esnt/uat2esnt.c -o uat2esnt/uat2esnt.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c uat2esnt/uat_decode.c -o uat2esnt/uat_decode.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c stats.c -o stats.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c cpr.c -o cpr.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c icao_filter.c -o icao_filter.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c track.c -o track.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c util.c -o util.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c fasthash.c -o fasthash.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c convert.c -o convert.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c sdr_ifile.c -o sdr_ifile.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c sdr_beast.c -o sdr_beast.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c sdr.c -o sdr.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c ais_charset.c -o ais_charset.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c globe_index.c -o globe_index.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c geomag.c -o geomag.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c receiver.c -o receiver.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c aircraft.c -o aircraft.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c api.c -o api.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c minilzo/minilzo.c -o minilzo.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c threadpool.c -o threadpool.o
cc -std=c11 -W -D_GNU_SOURCE -D_DEFAULT_SOURCE -Wall -Werror -fno-common -O2 -DMODES_READSB_VERSION=\""3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)"\" -Wdate-time -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Wformat -Werror=format-security -Wno-format-truncation -DSC16Q11_TABLE_BITS=8 -g  -c readsb.c -o readsb.o
cc -o readsb readsb.o argp.o anet.o interactive.o mode_ac.o mode_s.o comm_b.o json_out.o net_io.o crc.o demod_2400.o uat2esnt/uat2esnt.o uat2esnt/uat_decode.o stats.o cpr.o icao_filter.o track.o util.o fasthash.o convert.o sdr_ifile.o sdr_beast.o sdr.o ais_charset.o globe_index.o geomag.o receiver.o aircraft.o api.o minilzo.o threadpool.o  -pthread -lpthread -lm -lrt -lzstd -lz -lncurses
rm -f viewadsb
cp readsb viewadsb
root@raspberrypi:/usr/local/share/readsb# mkdir run
root@raspberrypi:/usr/local/share/readsb# ./readsb --net --devel=debugZstd --write-json run --quiet
invoked by: ./readsb --net --devel=debugZstd --write-json run --quiet
[2024-05-10 17:52:42.153 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)
calling ZSTD_createCCtx()
ZSTD_createCCtx() returned 0xf37005b8
calling ZSTD_compressCCtx() with cctx 0xf37005b8 pbuffer->size    196 src.buffer 0xf3700888 src.len    112 level 1 src.buffer[0] 0xbf cctx_first_byte 0x00
Bus error
root@raspberrypi:/usr/local/share/readsb# uname -a
Linux raspberrypi 5.10.103-v8+ #1529 SMP PREEMPT Tue Mar 8 12:26:46 GMT 2022 aarch64 GNU/Linux
root@raspberrypi:/usr/local/share/readsb# ldd readsb
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf7950000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xf7910000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xf788e000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xf7877000)
        libzstd.so.1 => /usr/lib/arm-linux-gnueabihf/libzstd.so.1 (0xf77de000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xf77b6000)
        libncurses.so.6 => /lib/arm-linux-gnueabihf/libncurses.so.6 (0xf7788000)
        libtinfo.so.6 => /lib/arm-linux-gnueabihf/libtinfo.so.6 (0xf7757000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xf760a000)
        /lib/ld-linux-armhf.so.3 (0xf7965000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xf75f7000)
root@raspberrypi:/usr/local/share/readsb#

md5sums 
cece1983e8e6e0afd2c492ee9ac8877d  /lib/arm-linux-gnueabihf/libpthread.so.0
8043b11e3ae9a08d09aa1fd81abc3cb3  /lib/arm-linux-gnueabihf/libm.so.6
a10579a6209cf01dd66709e0b28974e1  /lib/arm-linux-gnueabihf/librt.so.1
a90d4757f6a9281ed240f72b58c78d0e  /usr/lib/arm-linux-gnueabihf/libzstd.so.1
c4abbd13aeef0182c94b15a30d1fafbe  /lib/arm-linux-gnueabihf/libz.so.1
d360bb722010fb42bb9129c765a958a6  /lib/arm-linux-gnueabihf/libncurses.so.6
2cc245411ebe10811fee27eb1bb5976b  /lib/arm-linux-gnueabihf/libtinfo.so.6
662bff0b3e8c084e5893f8c47bd3cfb0  /lib/arm-linux-gnueabihf/libc.so.6
23a4a75712c54547249d565dbca3b942  /lib/ld-linux-armhf.so.3
589c28033d4a2646f7d53dee46c0bdca  /lib/arm-linux-gnueabihf/libdl.so.2

root@raspberrypi:/usr/local/share/readsb# apt update
Get:1 http://archive.raspberrypi.org/debian buster InRelease [32.6 kB]
Get:2 http://raspbian.raspberrypi.org/raspbian buster InRelease [15.0 kB]
Hit:3 http://repo.feed.flightradar24.com flightradar24 InRelease
Get:4 http://archive.raspberrypi.org/debian buster/main armhf Packages [400 kB]
Fetched 447 kB in 3s (149 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@raspberrypi:/usr/local/share/readsb# apt list --upgradable
Listing... Done
python-idna/oldoldstable 2.6-1+deb10u1 all [upgradable from: 2.6-1]
python3-idna/oldoldstable 2.6-1+deb10u1 all [upgradable from: 2.6-1]
root@raspberrypi:/usr/local/share/readsb#
wiedehopf commented 4 months ago 
gcc -v

This is Raspbian buster yes? I'm running the same thing on the same hardware. And i can't reproduce it. I don't know how to properly work with gdb to be honest. I've tried putting a break point ...

break ZSTD_compressCCtx

then running then stepping into the compression function. But really i'm out of my element here.

wiedehopf commented 4 months ago 
break util.c:881
run
... breakpoint gives you promt
si 50
.. repeat si 50 until you get the SIGBUS?

If you know better how to use gdb ... yeah. Not sure how this is gonna help either :/

I'm pretty sure the issue is not in my code.

Ok this is breaking gdb for me :) ....

wiedehopf commented 4 months ago

Ok different approach, i've made the destination Buffer be a multiple of 1024 ... Can't really change the source.

Anyhow maybe it just fixes it but i somehow doubt it.

The fact i can't reproduce it with the same software and hardware is really not good.

stefanpuch commented 4 months ago

gcc -v

root@raspberrypi:/usr/local/share/readsb# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/8/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Raspbian 8.3.0-6+rpi1' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=arm-linux-gnueabihf- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --disable-libquadmath-support --enable-plugin --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-sjlj-exceptions --with-arch=armv6 --with-fpu=vfp --with-float=hard --disable-werror --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1)
root@raspberrypi:/usr/local/share/readsb#

This is Raspbian buster yes?

cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
root@raspberrypi:/usr/local/share/readsb#

I don't know how to properly work with gdb to be honest. Is the same for me, but I tried and maybe we got a hint, see backttrace below


root@raspberrypi:/usr/local/share/readsb# gdb -q --args /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
Reading symbols from /usr/local/share/readsb/readsb...done.
(gdb) break util.c:881
Breakpoint 1 at 0x52bc8: file util.c, line 882.
(gdb) run
Starting program: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[2024-05-10 20:05:08.621 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: 9203c1c (committed: Fri May 10 13:48:08 2024 0200)
[New Thread 0xf7c4f440 (LWP 26636)]
[New Thread 0xf744e440 (LWP 26637)]
[New Thread 0xf6c4d440 (LWP 26638)]
[New Thread 0xf644c440 (LWP 26639)]
[New Thread 0xf5c4b440 (LWP 26640)]
[New Thread 0xf544a440 (LWP 26641)]
[New Thread 0xf4979440 (LWP 26642)]
[New Thread 0xf4178440 (LWP 26643)]
calling ZSTD_createCCtx()
ZSTD_createCCtx() returned 0xf38005b8
calling ZSTD_compressCCtx() with cctx 0xf38005b8 pbuffer->size    196 src.buffer 0xf3800888 src.len    112 level 1 src.buffer[0] 0xb4 cctx_first_byte 0x00
[New Thread 0xf35ff440 (LWP 26644)]
[Switching to Thread 0xf4178440 (LWP 26643)]

Thread 9 "readsb" hit Breakpoint 1, generateZstd (cctx=cctx@entry=0xf38005b8, pbuffer=0xf4177e0c, pbuffer@entry=0xf4177e04, src=..., level=level@entry=1) at util.c:882 882 size_t compressedSize = ZSTD_compressCCtx(cctx, (gdb) si 50

<3> readsb was suspended for more than 5 seconds, this isn't healthy you know! <3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/14027 ms, suppressing for 5 seconds! 0xf7fcda28 93 dl-runtime.c: No such file or directory. (gdb) si 50 <3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/6729 ms, suppressing for 5 seconds! 0xf7fc86a4 557 dl-lookup.c: No such file or directory. (gdb) si 50 0xf7fc86a4 557 in dl-lookup.c (gdb) bt #0 0xf7fc86a4 in dl_new_hash (s=0x118aa "sCCtx") at dl-lookup.c:557 #1 _dl_lookup_symbol_x (undef_name=0x1189e "ZSTD_compressCCtx", undef_map=undef_map@entry=0xf7fef978, ref=0xf4177d14, ref@entry=0xf4177d0c, symbol_scope=0x1, version=0x0, type_class=type_class@entry=1, flags=5, skip_map=skip_map@entry=0x0) at dl-lookup.c:792 #2 0xf7fcda80 in _dl_fixup (l=0xf7fef978, reloc_arg=) at dl-runtime.c:112 #3 0xf7fd3b6c in _dl_runtime_resolve () at ../sysdeps/arm/dl-trampoline.S:57 #4 0x00052bdc in generateZstd (cctx=cctx@entry=0xf38005b8, pbuffer=0xf4177e0c, pbuffer@entry=0xf4177e04, src=..., level=level@entry=1) at util.c:882 #5 0x00015980 in jsonEntryPoint (arg=) at readsb.c:598 #6 0xf7f6f494 in start_thread (arg=0xf4178440) at pthread_create.c:486 #7 0xf7d39e98 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6 Backtrace stopped: previous frame identical to this frame (corrupt stack?) ``` --> there is a "No such file or directory" in dl-lookup.c
wiedehopf commented 4 months ago

Can you

git pull and make -j4

Possibly the the multiple of 1024 actually helps ...

Otherwise you can try si 1000 until you get the error ...

Or just put this after the error instead of backtrace:

 x/256wa $sp
stefanpuch commented 4 months ago

Of cause, I pulled the updates (also did a make clean) recompiled si 1000 causes the debugger to stop

Starting program: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[2024-05-10 21:17:04.417 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: cc670c7 (committed: Fri May 10 19:20:25 2024 0200)
[New Thread 0xf7c4f440 (LWP 27289)]
[New Thread 0xf744e440 (LWP 27290)]
[New Thread 0xf6c4d440 (LWP 27291)]
[New Thread 0xf644c440 (LWP 27292)]
[New Thread 0xf5c4b440 (LWP 27293)]
[New Thread 0xf544a440 (LWP 27294)]
[New Thread 0xf4979440 (LWP 27295)]
[New Thread 0xf4178440 (LWP 27296)]
calling ZSTD_createCCtx()
ZSTD_createCCtx() returned 0xf38005b8
calling ZSTD_compressCCtx() with cctx 0xf38005b8 dstCapacity   1024 src.buffer 0xf3800888 src.len    112 level 1 src.buffer[0] 0x44 cctx_first_byte 0x00
[New Thread 0xf37ff440 (LWP 27297)]
[Switching to Thread 0xf4178440 (LWP 27296)]

Thread 9 "readsb" hit Breakpoint 1, generateZstd (cctx=cctx@entry=0xf38005b8, pbuffer=0xf4177e0c, pbuffer@entry=0xf4177e04, src=..., level=level@entry=1) at util.c:884
884         size_t compressedSize = ZSTD_compressCCtx(cctx,
(gdb) si 10000
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/6210 ms, suppressing for 5 seconds!

[3]+  Stopped                 gdb -q --args /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
root@raspberrypi:/usr/local/share/readsb#

Thus I tried to narrow to the SIGBUS

root@raspberrypi:/usr/local/share/readsb# gdb -q --args /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
Reading symbols from /usr/local/share/readsb/readsb...done.
(gdb) break util.c:881
Breakpoint 1 at 0x52bd4: file util.c, line 884.
(gdb) run
Starting program: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[2024-05-10 21:37:47.342 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: cc670c7 (committed: Fri May 10 19:20:25 2024 0200)
[New Thread 0xf7c4f440 (LWP 27460)]
[New Thread 0xf744e440 (LWP 27461)]
[New Thread 0xf6c4d440 (LWP 27462)]
[New Thread 0xf644c440 (LWP 27463)]
[New Thread 0xf5c4b440 (LWP 27464)]
[New Thread 0xf544a440 (LWP 27465)]
[New Thread 0xf4979440 (LWP 27466)]
[New Thread 0xf4178440 (LWP 27467)]
calling ZSTD_createCCtx()
ZSTD_createCCtx() returned 0xf2e005b8
[New Thread 0xf3977440 (LWP 27468)]
calling ZSTD_compressCCtx() with cctx 0xf2e005b8 dstCapacity   1024 src.buffer 0xf2e00888 src.len    112 level 1 src.buffer[0] 0x73 cctx_first_byte 0x00
[Switching to Thread 0xf4178440 (LWP 27467)]

Thread 9 "readsb" hit Breakpoint 1, generateZstd (cctx=cctx@entry=0xf2e005b8, pbuffer=0xf4177e0c, pbuffer@entry=0xf4177e04, src=..., level=level@entry=1) at util.c:884
884         size_t compressedSize = ZSTD_compressCCtx(cctx,
(gdb) break ZSTD_compress_usingDict
Breakpoint 2 at 0xf7e485a0
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) n
Program not restarted.
(gdb) c
Continuing.
<3> readsb was suspended for more than 5 seconds, this isn't healthy you know!
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/42238 ms, suppressing for 5 seconds!

Thread 9 "readsb" hit Breakpoint 2, 0xf7e485a0 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
<3> readsb was suspended for more than 5 seconds, this isn't healthy you know!
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/10414 ms, suppressing for 5 seconds!
0xf7e485a4 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485a8 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/2567 ms, suppressing for 5 seconds!
0xf7e485ac in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
<3>FATAL: priorityTasksRun() interval 60.6 seconds! Trying for an orderly shutdown as well as possible!
<3>lockThreads() probably hung on json
0xf7e485b0 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485b4 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/14761 ms, suppressing for 5 seconds!
0xf7e485b8 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485bc in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485c0 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485c4 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485c8 in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si
0xf7e485cc in ZSTD_compress_usingDict () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) c
Continuing.
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/2225 ms, suppressing for 5 seconds!

Thread 9 "readsb" received signal SIGBUS, Bus error.
0xf7e5430c in ?? () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) x/256wa $sp
0xf41778a8:     0x0     0xf2e008f0      0xf2e008f8      0x1
0xf41778b8:     0xf2e00887      0x1     0x0     0x18
0xf41778c8:     0xffffffe8      0x38    0xf41779a8      0xf2e0626c
0xf41778d8:     0xf2e07a78      0xf7e4275c      0x1     0x0
0xf41778e8:     0xf2e00888      0x0     0x4     0xf2e008f7
0xf41778f8:     0x0     0xf2e005b8      0xf7e53fcc      0xf2e007a8
0xf4177908:     0xf2e00888      0x70    0xf2e00730      0xf2e03a16
0xf4177918:     0xf2e03a19      0xf7e436c8      0x70    0x0
0xf4177928:     0x0     0x0     0x0     0x0
0xf4177938:     0x0     0x0     0x0     0x3f7
0xf4177948:     0x0     0xf2e03a19      0x0     0x90ed0 <__stack_chk_guard@@GLIBC_2.4>
0xf4177958:     0x0     0x0     0x0     0x0
0xf4177968:     0x0     0x0     0x0     0x0
0xf4177978:     0x0     0xf2e005b8      0xf4177b38      0x90ed0 <__stack_chk_guard@@GLIBC_2.4>
0xf4177988:     0x0     0x70    0x0     0xf2e005b8
0xf4177998:     0xf4177b38      0x90ed0 <__stack_chk_guard@@GLIBC_2.4>  0x0     0x70
0xf41779a8:     0x0     0xf7e42b08      0x0     0x0
0xf41779b8:     0x0     0xf2e06278      0x1     0x1
0xf41779c8:     0x0     0x0     0x3     0x0
0xf41779d8:     0x0     0x0     0x0     0x0
0xf41779e8:     0x0     0x0     0x0     0x0
0xf41779f8:     0x0     0x0     0x0     0x0
0xf4177a08:     0x0     0x0     0x70    0x0
0xf4177a18:     0x0     0x0     0x0     0x0
0xf4177a28:     0x0     0x0     0x0     0x0
0xf4177a38:     0x0     0x0     0x0     0x0
0xf4177a48:     0x0     0x0     0x0     0x0
0xf4177a58:     0x0     0x0     0x0     0x0
0xf4177a68:     0x0     0x0     0x0     0x0
0xf4177a78:     0x0     0x0     0x0     0x0
0xf4177a88:     0x0     0x0     0x0     0x0
0xf4177a98:     0x0     0x0     0x0     0x0
0xf4177aa8:     0x0     0x0     0x0     0x0
0xf4177ab8:     0x0     0x0     0x0     0x0
0xf4177ac8:     0x0     0x0     0x0     0x0
0xf4177ad8:     0x0     0x0     0x0     0x0
0xf4177ae8:     0x0     0x0     0x70    0x0
0xf4177af8:     0xf4177b80      0x90ed0 <__stack_chk_guard@@GLIBC_2.4>  0x0     0x9e3be200
0xf4177b08:     0x0     0x9e3be200      0xf2e005b8      0xf2e005b8
0xf4177b18:     0x70    0xf2e00888      0x3fa   0x70
0xf4177b28:     0xf2e008f8      0xf2e03a16      0xf2e03a19      0xf7e451f4
0xf4177b38:     0x70    0x0     0x0     0x8
0xf4177b48:     0x70    0x0     0x0     0x1
0xf4177b58:     0x1     0x6     0xf2e03a16      0x400
0xf4177b68:     0x1     0x0     0x0     0x0
0xf4177b78:     0x0     0xf4177c98      0xf2e005b8      0x70
0xf4177b88:     0x0     0x0     0x400   0xf2e03a10
0xf4177b98:     0xf2e00888      0xf7e456cc <ZSTD_compressEnd+44>        0x70    0x1
0xf4177ba8:     0x1     0x0     0x0     0x0
0xf4177bb8:     0xf4177c98      0xf4177c98      0xf4177c98      0xf2e005b8
0xf4177bc8:     0x70    0x0     0x0     0xf2e03a10
0xf4177bd8:     0x400   0xf7e48680 <ZSTD_compress_usingDict+224>        0x70    0x0
--Type <RET> for more, q to quit, c to continue without paging--
0xf4177be8:     0x0     0x0     0xa     0x7
0xf4177bf8:     0x8     0x1     0x5     0x0
0xf4177c08:     0x1     0x1     0x0     0x0
0xf4177c18:     0x3     0x0     0x0     0x0
0xf4177c28:     0x0     0x0     0x0     0x0
0xf4177c38:     0x0     0x0     0x0     0x0
0xf4177c48:     0x0     0x0     0x0     0x0
0xf4177c58:     0x1     0x1     0x0     0x0
0xf4177c68:     0xf4177cb4      0x90ed0 <__stack_chk_guard@@GLIBC_2.4>  0xa     0x7
0xf4177c78:     0x8     0x1     0x5     0x0
0xf4177c88:     0x1     0x1     0x0     0x0
0xf4177c98:     0x0     0xa     0x7     0x8

Looking at this I'm totally out... I can try to set a second breakpoint close before the ERROR

break ZSTD_getCParams

root@raspberrypi:/usr/local/share/readsb# gdb -q --args /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
Reading symbols from /usr/local/share/readsb/readsb...done.
(gdb) break util.c:881
Breakpoint 1 at 0x52bd4: file util.c, line 884.
(gdb) break ZSTD_getCParams
Function "ZSTD_getCParams" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (ZSTD_getCParams) pending.
(gdb) run
Starting program: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
invoked by: /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
[2024-05-10 21:45:25.375 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: cc670c7 (committed: Fri May 10 19:20:25 2024 0200)
[New Thread 0xf7c4f440 (LWP 27508)]
[New Thread 0xf744e440 (LWP 27509)]
[New Thread 0xf6c4d440 (LWP 27510)]
[New Thread 0xf644c440 (LWP 27511)]
[New Thread 0xf5c4b440 (LWP 27512)]
[New Thread 0xf544a440 (LWP 27513)]
[New Thread 0xf4979440 (LWP 27514)]
[New Thread 0xf4178440 (LWP 27515)]
calling ZSTD_createCCtx()
ZSTD_createCCtx() returned 0xf38005b8
calling ZSTD_compressCCtx() with cctx 0xf38005b8 dstCapacity   1024 src.buffer 0xf3800888 src.len    112 level 1 src.buffer[0] 0x9f cctx_first_byte 0x00
[New Thread 0xf37ff440 (LWP 27516)]
[Switching to Thread 0xf4178440 (LWP 27515)]

Thread 9 "readsb" hit Breakpoint 1, generateZstd (cctx=cctx@entry=0xf38005b8, pbuffer=0xf4177e0c, pbuffer@entry=0xf4177e04, src=..., level=level@entry=1) at util.c:884
884         size_t compressedSize = ZSTD_compressCCtx(cctx,
(gdb) c
Continuing.
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 1/0/0/2951 ms, suppressing for 5 seconds!

Thread 9 "readsb" hit Breakpoint 2, 0xf7e4703c in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/7228 ms, suppressing for 5 seconds!
0xf7e47050 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e47064 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e47078 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
<3>High load: modesNetPeriodicWork() elapsed1/2/3/interval 0/0/0/1206 ms, suppressing for 5 seconds!
0xf7e4708c in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e470a0 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e470b4 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e470c8 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e471b0 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e471c4 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e471d8 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e47100 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5
0xf7e47114 in ZSTD_getCParams () from /usr/lib/arm-linux-gnueabihf/libzstd.so.1
(gdb) si 5

[10]+  Stopped                 gdb -q --args /usr/local/share/readsb/readsb --net --devel=debugZstd --write-json /usr/local/share/readsb/run --quiet
root@raspberrypi:/usr/local/share/readsb#
wiedehopf commented 4 months ago

Yeah i'll just call this ... not my problem .. at this point.

You can easily find where ZSTD_compressCCtx is called in the source code and make modifications yourself.

Especially because i just can't reproduce it on my local system.

wiedehopf commented 4 months ago

Ok ... one more thing to try.

apt install --reinstall raspi-copies-and-fills libncurses6 libtinfo6 libc6 zlib1g libzstd1

If that doesn't fix it i'm really out of ideas. Had the realization that libzstd might also be using libraries that might be corrupted. These packages should reinstall all those libraries.

For later reference on how to find all the libraries:

for file in $(ldd readsb | cut -d= -f2 | cut -d\( -f1 | sed -e 's/^> //'); do dpkg -S $file; done

ldd will recursively check the program so it knows all the stuff the libraries load dynamically as well.

wiedehopf commented 4 months ago

Checked all the md5sums you already posted ... only one not yet checked:

 md5sum /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so
f625ac17fa92dfc2cb93948bb1e630b9  /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so

The others all seem to be matching.

wiedehopf commented 4 months ago 
root@raspberrypi:/usr/local/share/readsb# ldd readsb
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf7950000)

Oh almost missed that, somehow yours is point to v8

root@rpi3:~/readsb# ldd readsb
        linux-vdso.so.1 (0x7edac000)
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f88000)

while mine is pointing to v7

Did you post this yet?

uname -a
wiedehopf commented 4 months ago

Oh indeed you posted it and i missed it.

uname -a
Linux raspberrypi 5.10.103-v8+ #1529 SMP PREEMPT Tue Mar 8 12:26:46 GMT 2022 aarch64 GNU/Linux

So you're running 64bit kernel with 32bit userland This is ... pretty niche :)

I'll have to check if i can find an image that has that. At least then i can reproduce. But really not sure if i want to at this point.

Get rid of that cursed install :)

wiedehopf commented 4 months ago

The issue is that the SIGBUS doesn't happen in my software. It's in libzstd so even if we track it down i can't fix it. And even then ... you won't get a new package for your box.

So let's go with a workaround:

BRANCH="stale"
if grep -E 'wheezy|jessie' /etc/os-release -qs; then
    BRANCH="jessie"
fi

This branch jessie has a pretty old version of readsb that doesn't use libzstd. I suppose i could make the current version work without libzstd or just not use it dynamically.

I've pushed such a version to branch sid If you start it with --devel=disableZstd you should be able to work around this issue.

stefanpuch commented 4 months ago

OK, I did some more debugging and its defnitely related to libzstd. Maybe related to 64bit (I can check next) bu its working with the selfcompiled zstd 1.5.6

root@raspberrypi:/usr/local/share/readsb# ldd readsb
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf7d25000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xf7ce5000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xf7c63000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xf7c4c000)
        libzstd.so.1 => /usr/lib/arm-linux-gnueabihf/libzstd.so.1 (0xf7bb3000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xf7b8b000)
        libncurses.so.6 => /lib/arm-linux-gnueabihf/libncurses.so.6 (0xf7b5d000)
        libtinfo.so.6 => /lib/arm-linux-gnueabihf/libtinfo.so.6 (0xf7b2c000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xf79df000)
        /lib/ld-linux-armhf.so.3 (0xf7d3a000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xf79cc000)
root@raspberrypi:/usr/local/share/readsb# md5sum /usr/lib/arm-linux-gnueabihf/libzstd.so.1
a90d4757f6a9281ed240f72b58c78d0e  /usr/lib/arm-linux-gnueabihf/libzstd.so.1
root@raspberrypi:/usr/local/share/readsb# ls -l /usr/lib/arm-linux-gnueabihf/libzstd.so.1
lrwxrwxrwx 1 root root 16 May 11 10:24 /usr/lib/arm-linux-gnueabihf/libzstd.so.1 -> libzstd.so.1.3.8
root@raspberrypi:/usr/local/share/readsb# /usr/local/share/readsb/readsb --net  --write-json /usr/local/share/readsb/run --quiet
invoked by: /usr/local/share/readsb/readsb --net --write-json /usr/local/share/readsb/run --quiet
[2024-05-11 10:25:25.452 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: cc670c7-dirty (committed: Fri May 10 19:20:25 2024 0200)
Bus error
root@raspberrypi:/usr/local/share/readsb#

Now switching the zstd lib by chanking to dynamic link from /usr/lib/arm-linux-gnueabihf/libzstd.so.1 to the locally build one

root@raspberrypi:/usr/local/share/readsb# ldd readsb
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf7bac000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xf7b6c000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xf7aea000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0xf7ad3000)
        libzstd.so.1 => /usr/lib/arm-linux-gnueabihf/libzstd.so.1 (0xf79f5000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0xf79cd000)
        libncurses.so.6 => /lib/arm-linux-gnueabihf/libncurses.so.6 (0xf799f000)
        libtinfo.so.6 => /lib/arm-linux-gnueabihf/libtinfo.so.6 (0xf796e000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xf7821000)
        /lib/ld-linux-armhf.so.3 (0xf7bc1000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0xf780e000)
root@raspberrypi:/usr/local/share/readsb# md5sum /usr/lib/arm-linux-gnueabihf/libzstd.so.1
bdb122996d69a3f8de6c3c28cc0eebb8  /usr/lib/arm-linux-gnueabihf/libzstd.so.1
root@raspberrypi:/usr/local/share/readsb# ls -l /usr/lib/arm-linux-gnueabihf/libzstd.so.1
lrwxrwxrwx 1 root root 31 May 11 10:27 /usr/lib/arm-linux-gnueabihf/libzstd.so.1 -> /usr/local/lib/libzstd.so.1.5.6
root@raspberrypi:/usr/local/share/readsb# /usr/local/share/readsb/readsb --net  --write-json /usr/local/share/readsb/run --quiet
invoked by: /usr/local/share/readsb/readsb --net --write-json /usr/local/share/readsb/run --quiet
[2024-05-11 10:27:59.543 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: cc670c7-dirty (committed: Fri May 10 19:20:25 2024 0200)
^C[2024-05-11 10:28:25.236 CEST] Caught SIGINT, shutting down...
[2024-05-11 10:28:25.284 CEST] Normal exit.
wiedehopf commented 4 months ago

Ok ... i've pushed the workaround to the normal branches as well.

Just run the normal automatic install for readsb ... Add the option

--devel=disableZstd

And that should work around this weird corner case.

wiedehopf commented 4 months ago

It's related to running 32 bit armhf binaries with a 64bit aarch64 kernel. Otherwise i'd be able to reproduce this issue.

I wonder if you could fix it by somehow manipulating the PLATFORM thing.

        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f88000)

I don't know where that variable is set ... check man ldd For me it's v7l and for you it's v8l

And i don't think the rpi3 is a armv8 part so it's using instructions the processor doesn't support? Not 100% sure on that. (rpi3 is armv8 ... the 32bit system just runs an armv7 kernel)

wiedehopf commented 4 months ago 
$PLATFORM (or equivalently ${PLATFORM})
              This expands to a string corresponding to the processor
              type of the host system (e.g., "x86_64").  On some
              architectures, the Linux kernel doesn't provide a platform
              string to the dynamic linker.  The value of this string is
              taken from the AT_PLATFORM value in the auxiliary vector
              (see [getauxval(3)](https://man7.org/linux/man-pages/man3/getauxval.3.html)).

Yeah i doubt you can change that. Anyhow ... just use the workaround or stick to the self-compiled lizstd up to you.

stefanpuch commented 4 months ago

OK, now we have found some workarounds:

  1. I can use the --devel=disableZstd Switch than it do not crash anymore
  2. Its possible to change the dynamic link from /usr/lib/arm-linux-gnueabihf/libzstd.so.1 pointing to locally build version
  3. Its possile to change the Makefile to build readsb and replacing '-lzstd' with '/usr/lib/arm-linux-gnueabihf/libzstd.so.1' (where as last one is the locally build library which is linked statically

I'll also try to swich back to 32bit kernel. I switch some weeks ago from 32bit to 64bit by simply adding 'arm_64bit=1' into '/boot/config.txt'

Further mor I try to understand your last comments regarding reagrding PLATFORM thing

wiedehopf commented 4 months ago

The PLATFORM thing is just the library it links to. On 32bit kernel:

        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f88000)

On 64bit kernel:

        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v8l.so (0xf7bac000)

As the kernel itself really shouldn't matter, i believe that's the main difference. libzstd in that old version has a bug that causes issues only when loading the armv8 specific library.

stefanpuch commented 4 months ago

libzstd in that old version has a bug that causes issues only when loading the armv8 specific library. Seems to be true. I just did a reboot of my RPi where I commented out 'arm_64bit=1' in '/boot/config.txt' 

root@raspberrypi:/usr/local/share/readsb# uname -a
Linux raspberrypi 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux
root@raspberrypi:/usr/local/share/readsb# ldd readsb
        linux-vdso.so.1 (0x7ee40000)
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f62000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76f22000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76ea0000)
        librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76e89000)
        libzstd.so.1 => /usr/lib/arm-linux-gnueabihf/libzstd.so.1 (0x76df0000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x76dc8000)
        libncurses.so.6 => /lib/arm-linux-gnueabihf/libncurses.so.6 (0x76d9a000)
        libtinfo.so.6 => /lib/arm-linux-gnueabihf/libtinfo.so.6 (0x76d69000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76c1c000)
        /lib/ld-linux-armhf.so.3 (0x76f77000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x76c09000)
root@raspberrypi:/usr/local/share/readsb# /usr/local/share/readsb/readsb --net  --write-json /usr/local/share/readsb/run --quiet
invoked by: /usr/local/share/readsb/readsb --net --write-json /usr/local/share/readsb/run --quiet
[2024-05-11 11:11:54.357 CEST] readsb starting up.
readsb version: 3.14.1623 wiedehopf git: 3faa6d1-dirty (committed: Sat May 11 10:13:09 2024 0200)
^C[2024-05-11 11:11:56.285 CEST] Caught SIGINT, shutting down...
[2024-05-11 11:11:56.322 CEST] Normal exit.
root@raspberrypi:/usr/local/share/readsb#

Conclusion so far: "Raspbian GNU/Linux 10 (buster)" with 64bit kernel has a problem with libzstd

Option 1: stay on 32bit kernel Option 2: use the --devel=disableZstd switch (I don't know what side effects this has) Option 3: build libzstd from source (tested with 1.5.6) and link against that one by a) replacing the dynamic link b) link libzstd statically into readsb like menioned above

Many thanks to you for the effort tracking that down!