matrix-org / sytest

Black-box integration testing for Matrix homeservers
Apache License 2.0
72 stars 55 forks source link

install-deps.pl fails to build libsodium with gcc 11.2.1 #1114

Closed DMRobertson closed 2 years ago

DMRobertson commented 3 years ago

Describe the bug Unable to install sytest's perl dependencies following step 3 in README.md.

To Reproduce

  1. Clone sytest repo. Setup perl env vars as in the readme.
  2. ./install-deps.pl
  3. This fails when installing Alien::Sodium. There's a complaint when building libsodium about alignment.
make[3]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium'

[...]

  CC       crypto_generichash/blake2/ref/libsodium_la-blake2b-compress-ref.lo
In file included from crypto_generichash/blake2/ref/blake2b-compress-ref.c:5:
crypto_generichash/blake2/ref/blake2.h:118:5: error: size of array element is not a multiple of its alignment
  118 |     blake2s_state S[8][1];
      |     ^~~~~~~~~~~~~
crypto_generichash/blake2/ref/blake2.h:119:5: error: size of array element is not a multiple of its alignment
  119 |     blake2s_state R[1];
      |     ^~~~~~~~~~~~~
crypto_generichash/blake2/ref/blake2.h:126:5: error: size of array element is not a multiple of its alignment
  126 |     blake2b_state S[4][1];
      |     ^~~~~~~~~~~~~
crypto_generichash/blake2/ref/blake2.h:127:5: error: size of array element is not a multiple of its alignment
  127 |     blake2b_state R[1];
      |     ^~~~~~~~~~~~~
make[3]: *** [Makefile:2284: crypto_generichash/blake2/ref/libsodium_la-blake2b-compress-ref.lo] Error 1
Gory details ``` dmr on titan in sytest on  develop via 🐪 v5.32.1 2021-08-09 16:32:06 ✔ $ ./install-deps.pl **** install-deps.pl: Installing Alien::Sodium **** CPAN: Storable loaded ok (v3.21) Reading '/home/dmr/.local/share/.cpan/Metadata' Database was generated on Mon, 09 Aug 2021 14:41:03 GMT CPAN: Digest::SHA loaded ok (v6.02) CPAN: Compress::Zlib loaded ok (v2.102) Checksum for /home/dmr/.local/share/.cpan/sources/authors/id/A/AJ/AJGB/Alien-Sodium-1.0.8.0.tar.gz ok 'YAML' not installed, will not store persistent state CPAN: CPAN::Meta::Requirements loaded ok (v2.140) CPAN: Parse::CPAN::Meta loaded ok (v2.150010) CPAN: CPAN::Meta loaded ok (v2.150010) CPAN: Module::Build loaded ok (v0.4231) CPAN: Module::CoreList loaded ok (v5.20210620) Configuring A/AJ/AJGB/Alien-Sodium-1.0.8.0.tar.gz with Build.PL + pkg-config --exists 'libsodium = 1.0.8' && pkg-config --modversion libsodium Created MYMETA.yml and MYMETA.json Creating new 'Build' script for 'Alien-Sodium' version '1.0.8.0' AJGB/Alien-Sodium-1.0.8.0.tar.gz /usr/bin/perl Build.PL -- OK Running Build for A/AJ/AJGB/Alien-Sodium-1.0.8.0.tar.gz Building Alien-Sodium + pkg-config --exists 'libsodium = 1.0.8' && pkg-config --modversion libsodium Downloading File: libsodium-1.0.8.tar.gz ... Done Extracting Archive ... Done Building library ... + cd /home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8 + ./configure --with-pic --prefix=/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/blib/lib/auto/share/dist/Alien-Sodium --with-pic configure: loading site script /home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/config.site checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /usr/bin/mkdir -p checking for gawk... gawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether UID '1000' is supported by ustar format... yes checking whether GID '1000' is supported by ustar format... yes checking how to create a ustar tar archive... gnutar checking whether make supports nested variables... (cached) yes checking whether to enable maintainer-specific portions of Makefiles... no checking for style of include used by make... GNU checking for gcc... gcc checking whether the C compiler works... yes checking for C compiler default output file name... a.out checking for suffix of executables... checking whether we are cross compiling... no checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking whether gcc understands -c and -o together... yes checking dependency style of gcc... gcc3 checking for gcc option to accept ISO C99... none needed checking dependency style of gcc... gcc3 checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /usr/bin/grep checking for egrep... /usr/bin/grep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking minix/config.h usability... no checking minix/config.h presence... no checking for minix/config.h... no checking whether it is safe to define __EXTENSIONS__... yes checking for __native_client__ defined... no checking for _FORTIFY_SOURCE defined... yes checking whether C compiler accepts -fvisibility=hidden... yes checking whether C compiler accepts -fPIC... yes checking whether the linker accepts -fPIC... yes checking whether C compiler accepts -fPIE... yes checking whether the linker accepts -fPIE... yes checking whether the linker accepts -pie... yes checking whether C compiler accepts -fno-strict-aliasing... yes checking whether C compiler accepts -fno-strict-overflow... yes checking whether C compiler accepts -fstack-protector... yes checking whether the linker accepts -fstack-protector... yes checking whether C compiler accepts -Winit-self... yes checking whether C compiler accepts -Wwrite-strings... yes checking whether C compiler accepts -Wdiv-by-zero... yes checking whether C compiler accepts -Wsometimes-uninitialized... no checking whether C compiler accepts -Wall... yes checking whether C compiler accepts -Wall -Wextra... yes checking for clang... no checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init -Wparentheses... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init -Wparentheses -Wpointer-arith... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init -Wparentheses -Wpointer-arith -Wredundant-decls... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init -Wparentheses -Wpointer-arith -Wredundant-decls -Wstrict-prototypes... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init -Wparentheses -Wpointer-arith -Wredundant-decls -Wstrict-prototypes -Wswitch-enum... yes checking whether C compiler accepts -Wall -Wextra -Wbad-function-cast -Wcast-align -Wcast-qual -Wchar-subscripts -Wcomment -Wfloat-equal -Wformat=2 -Wimplicit -Wmissing-declarations -Wmissing-prototypes -Wnormalized=id -Woverride-init -Wparentheses -Wpointer-arith -Wredundant-decls -Wstrict-prototypes -Wswitch-enum -Wvariable-decl... no checking whether the linker accepts -Wl,-z,relro... yes checking whether the linker accepts -Wl,-z,now... yes checking whether the linker accepts -Wl,-z,noexecstack... yes checking how to print strings... printf checking for a sed that does not truncate output... /usr/bin/sed checking for fgrep... /usr/bin/grep -F checking for ld used by gcc... /usr/bin/ld checking if the linker (/usr/bin/ld) is GNU ld... yes checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B checking the name lister (/usr/bin/nm -B) interface... BSD nm checking whether ln -s works... yes checking the maximum length of command line arguments... 1572864 checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop checking for /usr/bin/ld option to reload object files... -r checking for objdump... objdump checking how to recognize dependent libraries... pass_all checking for dlltool... no checking how to associate runtime and link libraries... printf %s\n checking for ar... ar checking for archiver @FILE support... @ checking for strip... strip checking for ranlib... ranlib checking command to parse /usr/bin/nm -B output from gcc object... ok checking for sysroot... no checking for a working dd... /usr/bin/dd checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 checking for mt... no checking if : is a manifest tool... no checking for dlfcn.h... yes checking for objdir... .libs checking if gcc supports -fno-rtti -fno-exceptions... no checking for gcc option to produce PIC... -fPIC -DPIC checking if gcc PIC flag -fPIC -DPIC works... yes checking if gcc static flag -static works... no checking if gcc supports -c -o file.o... yes checking if gcc supports -c -o file.o... (cached) yes checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes checking whether -lc should be explicitly linked in... no checking dynamic linker characteristics... GNU/Linux ld.so checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes checking whether to build shared libraries... yes checking whether to build static libraries... yes checking for ar... (cached) ar checking for MMX instructions set... checking whether C compiler accepts -mmmx... yes yes checking whether C compiler accepts -mmmx... (cached) yes checking for SSE2 instructions set... checking whether C compiler accepts -msse2... yes yes checking whether C compiler accepts -msse2... (cached) yes checking whether C compiler accepts -msse3... yes checking for SSE3 instructions set... yes checking whether C compiler accepts -msse3... (cached) yes checking whether C compiler accepts -mssse3... yes checking for SSSE3 instructions set... yes checking whether C compiler accepts -mssse3... (cached) yes checking whether C compiler accepts -msse4.1... yes checking for SSE4.1 instructions set... yes checking whether C compiler accepts -msse4.1... (cached) yes checking whether C compiler accepts -mavx... yes checking for AVX instructions set... yes checking whether C compiler accepts -mavx... (cached) yes checking whether C compiler accepts -maes... yes checking whether C compiler accepts -mpclmul... yes checking for AESNI instructions set and PCLMULQDQ... yes checking whether C compiler accepts -maes... (cached) yes checking whether C compiler accepts -mpclmul... (cached) yes checking sys/mman.h usability... yes checking sys/mman.h presence... yes checking for sys/mman.h... yes checking for inline... inline checking whether byte ordering is bigendian... no checking whether __STDC_LIMIT_MACROS is required... no checking whether we can use x86_64 asm code... yes checking whether we can assemble AVX opcodes... yes checking for 128-bit arithmetic... yes checking for cpuid instruction... yes checking if weak symbols are supported... yes checking if data alignment is required... no checking for arc4random... no checking for arc4random_buf... no checking for mmap... yes checking for mlock... yes checking for madvise... yes checking for mprotect... yes checking for explicit_bzero... yes checking for posix_memalign... yes checking for getpid... yes checking if gcc/ld supports -Wl,--output-def... no checking that generated files are newer than configure... done configure: creating ./config.status config.status: creating Makefile config.status: creating dist-build/Makefile config.status: creating libsodium.pc config.status: creating libsodium-uninstalled.pc config.status: creating msvc-scripts/Makefile config.status: creating src/Makefile config.status: creating src/libsodium/Makefile config.status: creating src/libsodium/include/Makefile config.status: creating src/libsodium/include/sodium/version.h config.status: creating test/default/Makefile config.status: creating test/Makefile config.status: executing depfiles commands config.status: executing libtool commands + make Making all in dist-build make[1]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/dist-build' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/dist-build' Making all in msvc-scripts make[1]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/msvc-scripts' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/msvc-scripts' Making all in src make[1]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src' Making all in libsodium make[2]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium' Making all in include make[3]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium/include' make[3]: Nothing to be done for 'all'. make[3]: Leaving directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium/include' make[3]: Entering directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium' CC crypto_aead/chacha20poly1305/sodium/libsodium_la-aead_chacha20poly1305.lo CC crypto_auth/libsodium_la-crypto_auth.lo CC crypto_auth/hmacsha256/libsodium_la-auth_hmacsha256_api.lo CC crypto_auth/hmacsha256/cp/libsodium_la-hmac_hmacsha256.lo CC crypto_auth/hmacsha256/cp/libsodium_la-verify_hmacsha256.lo CC crypto_auth/hmacsha512/libsodium_la-auth_hmacsha512_api.lo CC crypto_auth/hmacsha512/cp/libsodium_la-hmac_hmacsha512.lo CC crypto_auth/hmacsha512/cp/libsodium_la-verify_hmacsha512.lo CC crypto_auth/hmacsha512256/libsodium_la-auth_hmacsha512256_api.lo CC crypto_auth/hmacsha512256/cp/libsodium_la-hmac_hmacsha512256.lo CC crypto_auth/hmacsha512256/cp/libsodium_la-verify_hmacsha512256.lo CC crypto_box/libsodium_la-crypto_box.lo CC crypto_box/libsodium_la-crypto_box_easy.lo CC crypto_box/libsodium_la-crypto_box_seal.lo CC crypto_box/curve25519xsalsa20poly1305/libsodium_la-box_curve25519xsalsa20poly1305_api.lo CC crypto_box/curve25519xsalsa20poly1305/ref/libsodium_la-after_curve25519xsalsa20poly1305.lo CC crypto_box/curve25519xsalsa20poly1305/ref/libsodium_la-before_curve25519xsalsa20poly1305.lo CC crypto_box/curve25519xsalsa20poly1305/ref/libsodium_la-box_curve25519xsalsa20poly1305.lo CC crypto_box/curve25519xsalsa20poly1305/ref/libsodium_la-keypair_curve25519xsalsa20poly1305.lo CC crypto_core/curve25519/ref10/libsodium_la-curve25519_ref10.lo CC crypto_core/hsalsa20/ref2/libsodium_la-core_hsalsa20.lo CC crypto_core/hsalsa20/libsodium_la-core_hsalsa20_api.lo CC crypto_core/salsa20/ref/libsodium_la-core_salsa20.lo CC crypto_core/salsa20/libsodium_la-core_salsa20_api.lo CC crypto_generichash/libsodium_la-crypto_generichash.lo CC crypto_generichash/blake2/libsodium_la-generichash_blake2_api.lo CC crypto_generichash/blake2/ref/libsodium_la-blake2b-compress-ref.lo In file included from crypto_generichash/blake2/ref/blake2b-compress-ref.c:5: crypto_generichash/blake2/ref/blake2.h:118:5: error: size of array element is not a multiple of its alignment 118 | blake2s_state S[8][1]; | ^~~~~~~~~~~~~ crypto_generichash/blake2/ref/blake2.h:119:5: error: size of array element is not a multiple of its alignment 119 | blake2s_state R[1]; | ^~~~~~~~~~~~~ crypto_generichash/blake2/ref/blake2.h:126:5: error: size of array element is not a multiple of its alignment 126 | blake2b_state S[4][1]; | ^~~~~~~~~~~~~ crypto_generichash/blake2/ref/blake2.h:127:5: error: size of array element is not a multiple of its alignment 127 | blake2b_state R[1]; | ^~~~~~~~~~~~~ make[3]: *** [Makefile:2284: crypto_generichash/blake2/ref/libsodium_la-blake2b-compress-ref.lo] Error 1 make[3]: Leaving directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium' make[2]: *** [Makefile:2967: all-recursive] Error 1 make[2]: Leaving directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src/libsodium' make[1]: Leaving directory '/home/dmr/.local/share/.cpan/build/Alien-Sodium-1.0.8.0-2/_alien/libsodium-1.0.8/src' make[1]: *** [Makefile:377: all-recursive] Error 1 make: *** [Makefile:491: all-recursive] Error 1 External command (make) failed! Error: 512 at ./Build line 64. Failed Build not completed at ./Build line 64. AJGB/Alien-Sodium-1.0.8.0.tar.gz ./Build -- NOT OK Stopping: 'install' failed for 'A/AJ/AJGB/Alien-Sodium-1.0.8.0.tar.gz'. Failed to import Alien::Sodium even after installing: unable to import Alien::Sodium: Can't locate Alien/Sodium.pm in @INC (you may need to install the Alien::Sodium module) (@INC contains: /home/dmr/lib/perl5/x86_64-linux-thread-multi /home/dmr/lib/perl5 /usr/local/lib64/perl5/5.32 /usr/local/share/perl5/5.32 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5). BEGIN failed--compilation aborted. ```

Expected behavior Successful installation.

Desktop (please complete the following information):

Additional context Discussed in various places:

GCC's behaviour apparently changed with https://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=50bc94898fac1bd9cc1dabf227208fb5d369c4c4

DMRobertson commented 3 years ago

I think this is now causing sytest:testing to fail to build.

See the action and a specific run: https://github.com/matrix-org/sytest/runs/4091959405?check_suite_focus=true#step:5:4467

DMRobertson commented 3 years ago

I'm guessing this happened after https://tracker.debian.org/news/1272843/gcc-defaults-1194-migrated-to-testing/ (the timing seems about right)

DMRobertson commented 3 years ago

Indeed:

gcc-defaults (1.192) experimental; urgency=medium

  * Default to GCC 11.

 -- Matthias Klose <doko@debian.org>  Sat, 23 Jan 2021 12:25:44 +0100
richvdh commented 3 years ago

I think we need to migrate away from Crypt::NaCl::Sodium, which seems to be abandonware.

https://metacpan.org/pod/Crypt::Ed25519 seems to be a good alternative, however to be able to drop Crypt::NaCl::Sodium altogether we need our dependencies (in particular Protocol::Matrix) to also drop it. To that end, I've filed https://rt.cpan.org/Public/Bug/Display.html?id=140079.

richvdh commented 2 years ago

looks like @leonerd doesn't have time to update Protocol::Matrix right now, so to avoid this being a blocker for us, let's just vendor my patched version in to sytest. (it's only 317 lines).