Clang sanitizer binaries segfault on CI

real-or-random commented 3 months ago

This is not related to a specific PR, the only reason why master is green is that it doesn't have recent pushes.

The first bad build is https://github.com/bitcoin-core/secp256k1/actions/runs/8317860245?pr=1058, which includes a rebuild of the Docker image... The last good build of the Docker image is https://github.com/bitcoin-core/secp256k1/actions/runs/8208332997/job/22451543872?pr=1479. But I'm not sure which difference caused the issue. There was no change on our side, so something must have changed in Debian. But what? Here are some things to look into:

[x] updating to clang-snapshot doesn't help, see #1505
[x] There was a recent clang packing fix related to the clang runtime, which looks suspicious: https://metadata.ftp-master.debian.org/changelogs//main/l/llvm-toolchain-14/llvm-toolchain-14_14.0.6-12_changelog, but the new version is apparently used already in the last good build
[ ] there's this related bug report, but it's two years old: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008309

We should probably diff the package lists (including package versions) from the CI outputs of the Docker image builds. This tells us what Debian packages got updated between the last good and the first bad build...

jonasnick commented 3 months ago

Interestingly the MuSig2 PR fails with a segfault in the tests and not in ./configure like the signed-digit multicomb PR.

I tried reproducing this locally using our docker file but failed: the tests pass in my local docker image.

I started the container with

docker build -t secp256k1-image -f ci/linux-debian.Dockerfile .
docker run -it --rm secp256k1-image

and then executed

git clone https://github.com/bitcoin-core/secp256k1.git
cd secp256k1
git fetch origin pull/1479/head:musig2-module
git checkout musig2-module
WERROR_CFLAGS="-Werror -pedantic-errors" MAKEFLAGS="-j4" BUILD="check" ECMULTWINDOW="auto" ECMULTGENPRECISION="auto" ASM="auto" WIDEMUL="auto" WITH_VALGRIND="yes" EXTRAFLAGS="" EXPERIMENTAL="no" ECDH="yes" RECOVERY="yes" SCHNORRSIG="yes" MUSIG="yes" ELLSWIFT="yes" SECP256K1_TEST_ITERS="32" BENCH="yes" SECP256K1_BENCH_ITERS="2" CTIMETESTS="no" EXAMPLES="yes" HOST="" WRAPPER_CMD="" CC="clang" CFLAGS="-fsanitize=undefined,address -g" ./ci/ci.sh

inside the container.

real-or-random commented 3 months ago

@jonasnick Can you try setting the additional sanitizer env variables that we set in the CI config? https://github.com/bitcoin-core/secp256k1/blob/427e86b9edcaa5cb19e71233ad93c2ec60dff10f/.github/workflows/ci.yml#L444-L446

Those should probably be added to our output for reproduction...

I'm just in the process of getting a diff of the Debian package versions.

real-or-random commented 3 months ago

I'm just in the process of getting a diff of the Debian package versions.

I double and triple checked that I used the right input, but believe it or not, the diff is empty...

For the record, this is the package list:

amd64 perl-modules-5.36 all 5.36.0-7+deb12u1
amd64 libgdbm6 amd64 1.23-3
amd64 libgdbm-compat4 amd64 1.23-3
amd64 libperl5.36 amd64 5.36.0-7+deb12u1
amd64 perl amd64 5.36.0-7+deb12u1
amd64 libssl3 amd64 3.0.11-1~deb12u2
amd64 libpython3.11-minimal amd64 3.11.2-6
amd64 libexpat1 amd64 2.5.0-1
amd64 python3.11-minimal amd64 3.11.2-6
amd64 python3-minimal amd64 3.11.2-1+b1
amd64 media-types all 10.0.0
amd64 libncursesw6 amd64 6.4-4
amd64 libkrb5support0 amd64 1.20.1-2+deb12u1
amd64 libk5crypto3 amd64 1.20.1-2+deb12u1
amd64 libkeyutils1 amd64 1.6.3-2
amd64 libkrb5-3 amd64 1.20.1-2+deb12u1
amd64 libgssapi-krb5-2 amd64 1.20.1-2+deb12u1
amd64 libtirpc-common all 1.3.3+ds-1
amd64 libtirpc3 amd64 1.3.3+ds-1
amd64 libnsl2 amd64 1.3.0-2
amd64 readline-common all 8.2-1.3
amd64 libreadline8 amd64 8.2-1.3
amd64 libsqlite3-0 amd64 3.40.1-2
amd64 libpython3.11-stdlib amd64 3.11.2-6
amd64 python3.11 amd64 3.11.2-6
amd64 libpython3-stdlib amd64 3.11.2-1+b1
amd64 python3 amd64 3.11.2-1+b1
amd64 bzip2 amd64 1.0.8-5+b1
amd64 openssl amd64 3.0.11-1~deb12u2
amd64 ca-certificates all 20230311
amd64 libmagic-mgc amd64 1:5.44-3
amd64 libmagic1 amd64 1:5.44-3
amd64 file amd64 1:5.44-3
amd64 xz-utils amd64 5.4.1-0.2
amd64 m4 amd64 1.4.19-3
amd64 autoconf all 2.71-3
amd64 autotools-dev all 20220109.1
amd64 automake all 1:1.16.5-1.3
amd64 binutils-common amd64 2.40-2
amd64 libbinutils amd64 2.40-2
amd64 libctf-nobfd0 amd64 2.40-2
amd64 libctf0 amd64 2.40-2
amd64 libgprofng0 amd64 2.40-2
amd64 libjansson4 amd64 2.14-2
amd64 binutils-x86-64-linux-gnu amd64 2.40-2
amd64 binutils amd64 2.40-2
amd64 binutils-arm-linux-gnueabihf amd64 2.40-2
amd64 binutils-mingw-w64-i686 amd64 2.40-2+10.4
amd64 binutils-mingw-w64-x86-64 amd64 2.40-2+10.4
amd64 binutils-powerpc64le-linux-gnu amd64 2.40-2
amd64 binutils-s390x-linux-gnu amd64 2.40-2
amd64 libbsd0 amd64 0.11.7-2
amd64 libedit2 amd64 3.1-20221030-2
amd64 libicu72 amd64 72.1-3
amd64 libxml2 amd64 2.9.14+dfsg-1.3~deb12u1
amd64 libz3-4 amd64 4.8.12-3.1
amd64 libllvm14 amd64 1:14.0.6-12
amd64 libclang-cpp14 amd64 1:14.0.6-12
amd64 libgomp1 amd64 12.2.0-14
amd64 libitm1 amd64 12.2.0-14
amd64 libatomic1 amd64 12.2.0-14
amd64 libasan8 amd64 12.2.0-14
amd64 liblsan0 amd64 12.2.0-14
amd64 libtsan2 amd64 12.2.0-14
amd64 libubsan1 amd64 12.2.0-14
amd64 libquadmath0 amd64 12.2.0-14
amd64 libgcc-12-dev amd64 12.2.0-14
amd64 libc-dev-bin amd64 2.36-9+deb12u4
amd64 linux-libc-dev amd64 6.1.76-1
amd64 libcrypt-dev amd64 1:4.4.33-2
amd64 libtirpc-dev amd64 1.3.3+ds-1
amd64 libnsl-dev amd64 1.3.0-2
amd64 rpcsvc-proto amd64 1.4.3-1
amd64 libc6-dev amd64 2.36-9+deb12u4
amd64 libstdc++-12-dev amd64 12.2.0-14
amd64 libgc1 amd64 1:8.2.2-3
amd64 libobjc4 amd64 12.2.0-14
amd64 libobjc-12-dev amd64 12.2.0-14
amd64 libclang-common-14-dev all 1:14.0.6-12
amd64 llvm-14-linker-tools amd64 1:14.0.6-12
amd64 libclang1-14 amd64 1:14.0.6-12
amd64 clang-14 amd64 1:14.0.6-12
amd64 clang amd64 1:14.0-55.7~deb12u1
amd64 libisl23 amd64 0.25-1.1
amd64 libmpfr6 amd64 4.2.0-1
amd64 libmpc3 amd64 1.3.1-1
amd64 cpp-12 amd64 12.2.0-14
amd64 cpp amd64 4:12.2.0-3
amd64 gcc-12-arm-linux-gnueabihf-base amd64 12.2.0-14cross1
amd64 cpp-12-arm-linux-gnueabihf amd64 12.2.0-14cross1
amd64 gcc-12-i686-linux-gnu-base amd64 12.2.0-14cross1
amd64 cpp-12-i686-linux-gnu amd64 12.2.0-14cross1
amd64 gcc-12-powerpc64le-linux-gnu-base amd64 12.2.0-14cross1
amd64 cpp-12-powerpc64le-linux-gnu amd64 12.2.0-14cross1
amd64 gcc-12-s390x-linux-gnu-base amd64 12.2.0-14cross1
amd64 cpp-12-s390x-linux-gnu amd64 12.2.0-14cross1
amd64 cpp-arm-linux-gnueabihf amd64 4:12.2.0-3
amd64 cpp-i686-linux-gnu amd64 4:12.2.0-3
amd64 cpp-powerpc64le-linux-gnu amd64 4:12.2.0-3
amd64 cpp-s390x-linux-gnu amd64 4:12.2.0-3
amd64 libdpkg-perl all 1.21.22
amd64 patch amd64 2.7.6-7
amd64 make amd64 4.3-4.1
amd64 dpkg-dev all 1.21.22
amd64 fonts-dejavu-core all 2.37-6
amd64 fontconfig-config amd64 2.14.1-4
amd64 libcc1-0 amd64 12.2.0-14
amd64 gcc-12 amd64 12.2.0-14
amd64 gcc amd64 4:12.2.0-3
amd64 g++-12 amd64 12.2.0-14
amd64 g++ amd64 4:12.2.0-3
amd64 gcc-12-cross-base all 12.2.0-14cross1
amd64 libc6-armhf-cross all 2.36-8cross1
amd64 libgcc-s1-armhf-cross all 12.2.0-14cross1
amd64 libgomp1-armhf-cross all 12.2.0-14cross1
amd64 libatomic1-armhf-cross all 12.2.0-14cross1
amd64 libasan8-armhf-cross all 12.2.0-14cross1
amd64 libstdc++6-armhf-cross all 12.2.0-14cross1
amd64 libubsan1-armhf-cross all 12.2.0-14cross1
amd64 libgcc-12-dev-armhf-cross all 12.2.0-14cross1
amd64 gcc-12-arm-linux-gnueabihf amd64 12.2.0-14cross1
armhf gcc-12-base armhf 12.2.0-14
s390x gcc-12-base s390x 12.2.0-14
i386 gcc-12-base i386 12.2.0-14
ppc64el gcc-12-base ppc64el 12.2.0-14
amd64 binutils-i686-linux-gnu amd64 2.40-2
amd64 libc6-i386-cross all 2.36-8cross1
amd64 libgcc-s1-i386-cross all 12.2.0-14cross1
amd64 libgomp1-i386-cross all 12.2.0-14cross1
amd64 libitm1-i386-cross all 12.2.0-14cross1
amd64 libatomic1-i386-cross all 12.2.0-14cross1
amd64 libasan8-i386-cross all 12.2.0-14cross1
amd64 libstdc++6-i386-cross all 12.2.0-14cross1
amd64 libubsan1-i386-cross all 12.2.0-14cross1
amd64 libquadmath0-i386-cross all 12.2.0-14cross1
amd64 libgcc-12-dev-i386-cross all 12.2.0-14cross1
amd64 gcc-12-i686-linux-gnu amd64 12.2.0-14cross1
amd64 libc6-ppc64el-cross all 2.36-8cross1
amd64 libgcc-s1-ppc64el-cross all 12.2.0-14cross1
amd64 libgomp1-ppc64el-cross all 12.2.0-14cross1
amd64 libitm1-ppc64el-cross all 12.2.0-14cross1
amd64 libatomic1-ppc64el-cross all 12.2.0-14cross1
amd64 libasan8-ppc64el-cross all 12.2.0-14cross1
amd64 liblsan0-ppc64el-cross all 12.2.0-14cross1
amd64 libtsan2-ppc64el-cross all 12.2.0-14cross1
amd64 libstdc++6-ppc64el-cross all 12.2.0-14cross1
amd64 libubsan1-ppc64el-cross all 12.2.0-14cross1
amd64 libquadmath0-ppc64el-cross all 12.2.0-14cross1
amd64 libgcc-12-dev-ppc64el-cross all 12.2.0-14cross1
amd64 gcc-12-powerpc64le-linux-gnu amd64 12.2.0-14cross1
amd64 libc6-s390x-cross all 2.36-8cross1
amd64 libgcc-s1-s390x-cross all 12.2.0-14cross1
amd64 libgomp1-s390x-cross all 12.2.0-14cross1
amd64 libitm1-s390x-cross all 12.2.0-14cross1
amd64 libatomic1-s390x-cross all 12.2.0-14cross1
amd64 libasan8-s390x-cross all 12.2.0-14cross1
amd64 libstdc++6-s390x-cross all 12.2.0-14cross1
amd64 libubsan1-s390x-cross all 12.2.0-14cross1
amd64 libgcc-12-dev-s390x-cross all 12.2.0-14cross1
amd64 gcc-12-s390x-linux-gnu amd64 12.2.0-14cross1
amd64 gcc-arm-linux-gnueabihf amd64 4:12.2.0-3
amd64 gcc-i686-linux-gnu amd64 4:12.2.0-3
amd64 gcc-mingw-w64-base amd64 12.2.0-14+25.2
amd64 mingw-w64-common all 10.0.0-3
amd64 mingw-w64-i686-dev all 10.0.0-3
amd64 gcc-mingw-w64-i686-win32-runtime amd64 12.2.0-14+25.2
amd64 gcc-mingw-w64-i686-win32 amd64 12.2.0-14+25.2
amd64 mingw-w64-x86-64-dev all 10.0.0-3
amd64 gcc-mingw-w64-x86-64-win32-runtime amd64 12.2.0-14+25.2
amd64 gcc-mingw-w64-x86-64-win32 amd64 12.2.0-14+25.2
amd64 gcc-powerpc64le-linux-gnu amd64 4:12.2.0-3
amd64 gcc-s390x-linux-gnu amd64 4:12.2.0-3
amd64 libbrotli1 amd64 1.0.9-2+b6
amd64 libsasl2-modules-db amd64 2.1.28+dfsg-10
amd64 libsasl2-2 amd64 2.1.28+dfsg-10
amd64 libldap-2.5-0 amd64 2.5.13+dfsg-5
amd64 libnghttp2-14 amd64 1.52.0-1+deb12u1
amd64 libpsl5 amd64 0.21.2-1
amd64 librtmp1 amd64 2.4+20151223.gitfa8646d.1-2+b2
amd64 libssh2-1 amd64 1.10.0-3+b1
amd64 libcurl3-gnutls amd64 7.88.1-10+deb12u5
amd64 liberror-perl all 0.17029-2
amd64 git-man all 1:2.39.2-1.1
amd64 git amd64 1:2.39.2-1.1
amd64 iso-codes all 4.15.0-1
amd64 libc6-i386 amd64 2.36-9+deb12u4
amd64 lib32gcc-s1 amd64 12.2.0-14
amd64 lib32stdc++6 amd64 12.2.0-14
amd64 libabsl20220623 amd64 20220623.1-1
i386 libgcc-s1 i386 12.2.0-14
i386 libc6 i386 2.36-9+deb12u4
i386 libstdc++6 i386 12.2.0-14
i386 libabsl20220623 i386 20220623.1-1
i386 libaom3 i386 3.6.0-1
amd64 libaom3 amd64 3.6.0-1
i386 libasan8 i386 12.2.0-14
amd64 libasound2-data all 1.2.8-1
i386 libasound2 i386 1.2.8-1+b1
amd64 libasound2 amd64 1.2.8-1+b1
amd64 libasyncns0 amd64 0.8-6+b3
i386 libasyncns0 i386 0.8-6+b3
amd64 libdav1d6 amd64 1.0.0-2
amd64 libgav1-1 amd64 0.18.0-1+b1
amd64 librav1e0 amd64 0.5.1-6
amd64 libsvtav1enc1 amd64 1.4.1+dfsg-1
amd64 libjpeg62-turbo amd64 1:2.1.5-2
amd64 libyuv0 amd64 0.0~git20230123.b2528b0-1
amd64 libavif15 amd64 0.11.1-1
i386 libdav1d6 i386 1.0.0-2
i386 libgav1-1 i386 0.18.0-1+b1
i386 librav1e0 i386 0.5.1-6
i386 libjpeg62-turbo i386 1:2.1.5-2
i386 libyuv0 i386 0.0~git20230123.b2528b0-1
i386 libavif15 i386 0.11.1-1
i386 libblkid1 i386 2.38.1-5+b1
i386 libbrotli1 i386 1.0.9-2+b6
i386 libmd0 i386 1.0.4-2
i386 libbsd0 i386 0.11.7-2
i386 libbz2-1.0 i386 1.0.8-5+b1
ppc64el libgcc-s1 ppc64el 12.2.0-14
ppc64el libc6 ppc64el 2.36-9+deb12u4
s390x libgcc-s1 s390x 12.2.0-14
s390x libc6 s390x 2.36-9+deb12u4
armhf libgcc-s1 armhf 12.2.0-14
armhf libc6 armhf 2.36-9+deb12u4
ppc64el libc6-dbg ppc64el 2.36-9+deb12u4
amd64 libc6-dbg amd64 2.36-9+deb12u4
armhf libc6-dbg armhf 2.36-9+deb12u4
s390x libc6-dbg s390x 2.36-9+deb12u4
i386 libc6-dbg i386 2.36-9+deb12u4
amd64 linux-libc-dev-armhf-cross all 6.1.4-1cross1
amd64 libc6-dev-armhf-cross all 2.36-8cross1
amd64 linux-libc-dev-i386-cross all 6.1.4-1cross1
amd64 libc6-dev-i386-cross all 2.36-8cross1
amd64 linux-libc-dev-ppc64el-cross all 6.1.4-1cross1
amd64 libc6-dev-ppc64el-cross all 2.36-8cross1
amd64 linux-libc-dev-s390x-cross all 6.1.4-1cross1
amd64 libc6-dev-s390x-cross all 2.36-8cross1
i386 libcap2 i386 1:2.66-4
amd64 libcap2-bin amd64 1:2.66-4
amd64 libcapi20-3 amd64 1:3.27-3+b1
i386 libcapi20-3 i386 1:3.27-3+b1
amd64 libcapstone4 amd64 4.0.2-5
amd64 libclang-rt-14-dev amd64 1:14.0.6-12
amd64 libclang-rt-dev amd64 1:14.0-55.7~deb12u1
i386 libcom-err2 i386 1.47.0-2
amd64 libnspr4 amd64 2:4.35-1
amd64 libnss3 amd64 2:3.87.1-1
amd64 nss-plugin-pem amd64 1.0.8+1-1
amd64 libcurl3-nss amd64 7.88.1-10+deb12u5
amd64 libcurl4 amd64 7.88.1-10+deb12u5
i386 libkrb5support0 i386 1.20.1-2+deb12u1
i386 libk5crypto3 i386 1.20.1-2+deb12u1
i386 libkeyutils1 i386 1.6.3-2
i386 libssl3 i386 3.0.11-1~deb12u2
i386 libkrb5-3 i386 1.20.1-2+deb12u1
i386 libgssapi-krb5-2 i386 1.20.1-2+deb12u1
i386 libunistring2 i386 1.0-2
i386 libidn2-0 i386 2.3.3-1+b1
i386 libgmp10 i386 2:6.2.1+dfsg1-1.1
i386 libnettle8 i386 3.8.1-2
i386 libhogweed6 i386 3.8.1-2
i386 libffi8 i386 3.4.4-1
i386 libp11-kit0 i386 0.24.1-2
i386 libtasn1-6 i386 4.19.0-2
i386 libgnutls30 i386 3.7.9-2+deb12u2
i386 libdb5.3 i386 5.3.28+dfsg2-1
i386 libsasl2-modules-db i386 2.1.28+dfsg-10
i386 libsasl2-2 i386 2.1.28+dfsg-10
i386 libldap-2.5-0 i386 2.5.13+dfsg-5
i386 libnghttp2-14 i386 1.52.0-1+deb12u1
i386 libpsl5 i386 0.21.2-1
i386 zlib1g i386 1:1.2.13.dfsg-1
i386 librtmp1 i386 2.4+20151223.gitfa8646d.1-2+b2
i386 libssh2-1 i386 1.10.0-3+b1
i386 libzstd1 i386 1.5.4+dfsg2-5
i386 libcurl4 i386 7.88.1-10+deb12u5
amd64 libdbus-1-3 amd64 1.14.10-1~deb12u1
i386 libgpg-error0 i386 1.46-1
i386 libgcrypt20 i386 1.10.1-3
i386 liblz4-1 i386 1.9.4-1
i386 liblzma5 i386 5.4.1-0.2
i386 libsystemd0 i386 252.22-1~deb12u1
i386 libdbus-1-3 i386 1.14.10-1~deb12u1
i386 libde265-0 i386 1.0.11-1+deb12u2
amd64 libde265-0 amd64 1.0.11-1+deb12u2
i386 libdeflate0 i386 1.14-1
amd64 libdeflate0 amd64 1.14-1
amd64 libelf1 amd64 0.188-2.1
amd64 libdw1 amd64 0.188-2.1
i386 libelf1 i386 0.188-2.1
i386 libdw1 i386 0.188-2.1
i386 libexif12 i386 0.6.24-1+b1
amd64 libexif12 amd64 0.6.24-1+b1
i386 libexpat1 i386 2.5.0-1
i386 libogg0 i386 1.3.5-3
i386 libflac12 i386 1.4.2+ds-2
amd64 libogg0 amd64 1.3.5-3
amd64 libflac12 amd64 1.4.2+ds-2
i386 libpng16-16 i386 1.6.39-2
i386 libfreetype6 i386 2.12.1+dfsg-5
i386 libfontconfig1 i386 2.14.1-4
amd64 libpng16-16 amd64 1.6.39-2
amd64 libfreetype6 amd64 2.12.1+dfsg-5
amd64 libfontconfig1 amd64 2.14.1-4
i386 libnuma1 i386 2.0.16-1
i386 libx265-199 i386 3.5-2+b1
i386 libheif1 i386 1.15.1-1
i386 libjbig0 i386 2.1-6.1
i386 liblerc4 i386 4.0.0+ds-2
i386 libwebp7 i386 1.2.4-0.2+deb12u1
i386 libtiff6 i386 4.5.0-6+deb12u1
i386 libxau6 i386 1:1.0.9-1
i386 libxdmcp6 i386 1:1.1.2-3
i386 libxcb1 i386 1.15-1
amd64 libx11-data all 2:1.8.4-2+deb12u2
i386 libx11-6 i386 2:1.8.4-2+deb12u2
i386 libxpm4 i386 1:3.5.12-1.1+deb12u1
i386 libgd3 i386 2.3.3-9
amd64 libnuma1 amd64 2.0.16-1
amd64 libx265-199 amd64 3.5-2+b1
amd64 libheif1 amd64 1.15.1-1
amd64 libjbig0 amd64 2.1-6.1
amd64 liblerc4 amd64 4.0.0+ds-2
amd64 libwebp7 amd64 1.2.4-0.2+deb12u1
amd64 libtiff6 amd64 4.5.0-6+deb12u1
amd64 libxau6 amd64 1:1.0.9-1
amd64 libxdmcp6 amd64 1:1.1.2-3
amd64 libxcb1 amd64 1.15-1
amd64 libx11-6 amd64 2:1.8.4-2+deb12u2
amd64 libxpm4 amd64 1:3.5.12-1.1+deb12u1
amd64 libgd3 amd64 2.3.3-9
i386 libpcre2-8-0 i386 10.42-1
i386 libselinux1 i386 3.4-1+b6
i386 libmount1 i386 2.38.1-5+b1
i386 libglib2.0-0 i386 2.74.6-2
amd64 libglib2.0-0 amd64 2.74.6-2
i386 libltdl7 i386 2.4.7-5
i386 libudev1 i386 252.22-1~deb12u1
i386 libusb-1.0-0 i386 2:1.0.26-1
i386 libgphoto2-port12 i386 2.5.30-1
i386 libicu72 i386 72.1-3
i386 libxml2 i386 2.9.14+dfsg-1.3~deb12u1
i386 libgphoto2-6 i386 2.5.30-1
amd64 libltdl7 amd64 2.4.7-5
amd64 libusb-1.0-0 amd64 2:1.0.26-1
amd64 libgphoto2-port12 amd64 2.5.30-1
amd64 libgphoto2-6 amd64 2.5.30-1
i386 libunwind8 i386 1.6.2-3
i386 libgstreamer1.0-0 i386 1.22.0-2
i386 liborc-0.4-0 i386 1:0.4.33-2
i386 libgstreamer-plugins-base1.0-0 i386 1.22.0-3+deb12u1
amd64 libunwind8 amd64 1.6.2-3
amd64 libgstreamer1.0-0 amd64 1.22.0-2
amd64 liborc-0.4-0 amd64 1:0.4.33-2
amd64 libgstreamer-plugins-base1.0-0 amd64 1.22.0-3+deb12u1
amd64 libmp3lame0 amd64 3.100-6
i386 libmp3lame0 i386 3.100-6
i386 libmpg123-0 i386 1.31.2-1
amd64 libmpg123-0 amd64 1.31.2-1
amd64 libopus0 amd64 1.3.1-3
i386 libopus0 i386 1.3.1-3
amd64 libpcap0.8 amd64 1.10.3-1
i386 libpcap0.8 i386 1.10.3-1
amd64 libpfm4 amd64 4.13.0-1
amd64 libpkgconf3 amd64 1.8.1-1
amd64 libvorbis0a amd64 1.3.7-1
amd64 libvorbisenc2 amd64 1.3.7-1
amd64 libsndfile1 amd64 1.2.0-1
amd64 libx11-xcb1 amd64 2:1.8.4-2+deb12u2
amd64 libpulse0 amd64 16.1+dfsg1-2+b1
i386 libvorbis0a i386 1.3.7-1
i386 libvorbisenc2 i386 1.3.7-1
i386 libsndfile1 i386 1.2.0-1
i386 libx11-xcb1 i386 2:1.8.4-2+deb12u2
i386 libpulse0 i386 16.1+dfsg1-2+b1
amd64 libtool all 2.4.7-5
i386 libubsan1 i386 12.2.0-14
amd64 liburing2 amd64 2.3-3
i386 libxext6 i386 2:1.3.4-1+b1
i386 ocl-icd-libopencl1 i386 2.3.1-1
amd64 libz-mingw-w64 all 1.2.13+dfsg-1
i386 libwine i386 8.0~repack-4
amd64 libxext6 amd64 2:1.3.4-1+b1
amd64 ocl-icd-libopencl1 amd64 2.3.1-1
amd64 libwine amd64 8.0~repack-4
amd64 llvm-14-runtime amd64 1:14.0.6-12
amd64 llvm-runtime amd64 1:14.0-55.7~deb12u1
amd64 llvm-14 amd64 1:14.0.6-12
amd64 llvm amd64 1:14.0-55.7~deb12u1
amd64 pkgconf-bin amd64 1.8.1-1
amd64 pkgconf amd64 1.8.1-1
amd64 pkg-config amd64 1.8.1-1
amd64 qemu-user amd64 1:7.2+dfsg-7+deb12u5
amd64 valgrind amd64 1:3.19.0-1
amd64 wine64 amd64 8.0~repack-4
i386 wine32 i386 8.0~repack-4
amd64 wine all 8.0~repack-4

there's this related bug report, but it's two years old: bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008309

I'm starting to believe that this is related. It mentions some sysctl settings, and searching for them gives me https://github.com/google/sanitizers/issues/1614 with a bunch of linked issues in the past days... We should probably try (sudo) sysctl vm.mmap_rnd_bits=28.

jonasnick commented 3 months ago

Can you try setting the additional sanitizer env variables that we set in the CI config?

Good point. Still 0 crashes in 10 runs of

UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1" ASAN_OPTIONS="strict_string_checks=1:detect_stack_use_after_return=1:detect_leaks=1" LSAN_OPTIONS="use_unaligned=1" WERROR_CFLAGS="-Werror -pedantic-errors" MAKEFLAGS="-j4" BUILD="check" ECMULTWINDOW="auto" ECMULTGENPRECISION="auto" ASM="auto" WIDEMUL="auto" WITH_VALGRIND="yes" EXTRAFLAGS="" EXPERIMENTAL="no" ECDH="yes" RECOVERY="yes" SCHNORRSIG="yes" MUSIG="yes" ELLSWIFT="yes" SECP256K1_TEST_ITERS="32" BENCH="yes" SECP256K1_BENCH_ITERS="2" CTIMETESTS="no" EXAMPLES="yes" HOST="" WRAPPER_CMD="" CC="clang" CFLAGS="-fsanitize=undefined,address -g" ./ci/ci.sh

jonasnick commented 3 months ago

I can reproduce this now after looking into the bug reports. Need to set both

vm.mmap_rnd_bits = 32
vm.mmap_rnd_compat_bits = 16

on the host for the crashes to happen.

fanquake commented 3 months ago

Note that the upstream issue should now be fixed: https://github.com/actions/runner-images/issues/9491 (see also https://github.com/google/oss-fuzz/issues/11703).

real-or-random commented 3 months ago

Hm, okay, and this is their fix (it's the same as our workaround): https://github.com/actions/runner-images/pull/9513/commits/7aba0ab8abda5860b693ed43d74241a652660356

I'd tend to keep the workaround in our code base. I think it's good to have it for documentation purposes. We could maybe expand a bit on the comment, and explain that this will be resolved in future clang releases.

real-or-random commented 3 months ago

We could maybe expand a bit on the comment, and explain that this will be resolved in future clang releases.

Let's keep this open until https://github.com/llvm/llvm-project/commit/58f7251820b14c93168726a24816d8a094599be5 is in a LLVM release (check branch info displayed by GitHub or the table at https://github.com/google/sanitizers/issues/1614#issuecomment-2010316781). Then we can change the comment to say that the workaround can be removed once we move to that clang version.

fanquake commented 3 months ago

Let's keep this open until https://github.com/llvm/llvm-project/commit/58f7251820b14c93168726a24816d8a094599be5 is in a LLVM release

Should be backported soon: https://github.com/llvm/llvm-project/pull/86201.

thurstond commented 3 months ago

Let's keep this open until llvm/llvm-project@58f7251 is in a LLVM release

Should be backported soon: llvm/llvm-project#86201.

Backport has landed in the 18.x branch (release pending): https://github.com/llvm/llvm-project/commit/c2a57034eff048cd36c563c8e0051db3a70991b3

real-or-random commented 3 months ago

@thurstond Thanks for the explicit notification!

fanquake commented 3 months ago

LLVM 18.1.3 is out, which contains the fix: https://github.com/llvm/llvm-project/releases/tag/llvmorg-18.1.3.

bitcoin-core / secp256k1

Clang sanitizer binaries segfault on CI #1506