ebiggers / libdeflate

Heavily optimized library for DEFLATE/zlib/gzip compression and decompression
MIT License
997 stars 165 forks source link

Compilation Error Despite Disabling AVX Instructions #386

Closed shreepo closed 1 week ago

shreepo commented 3 weeks ago

I am encountering a persistent issue during the compilation of a project where the compiler seems to be ignoring directives to disable AVX, AVX2, and AVX-512 instructions. Despite specifying flags to prevent these instructions, the compiler attempts to use AVX-512 related instructions such as vpdpbusd, resulting in errors because these are not recognized by the assembler.

Environment:

Compiler: GCC 9.3.0 Operating System: Centos 7.9 CMake Version: 3.28.1

I configured the project with the following CMake command: cmake .. -DCMAKE_INSTALL_PREFIX=~/local/libdeflate -DCMAKE_C_FLAGS="-march=native -mno-avx -mno-avx2 -mno-avx512f" - No build type selected; defaulting to Release -- The C compiler identification is GNU 9.3.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /home/chongchuanlong/.local/gcc-9.3.0/bin/gcc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Performing Test HAVE_WDECLARATION_AFTER_STATEMENT -- Performing Test HAVE_WDECLARATION_AFTER_STATEMENT - Success -- Performing Test HAVE_WIMPLICIT_FALLTHROUGH -- Performing Test HAVE_WIMPLICIT_FALLTHROUGH - Success -- Performing Test HAVE_WMISSING_FIELD_INITIALIZERS -- Performing Test HAVE_WMISSING_FIELD_INITIALIZERS - Success -- Performing Test HAVE_WMISSING_PROTOTYPES -- Performing Test HAVE_WMISSING_PROTOTYPES - Success -- Performing Test HAVE_WPEDANTIC -- Performing Test HAVE_WPEDANTIC - Success -- Performing Test HAVE_WSHADOW -- Performing Test HAVE_WSHADOW - Success -- Performing Test HAVE_WSTRICT_PROTOTYPES -- Performing Test HAVE_WSTRICT_PROTOTYPES - Success -- Performing Test HAVE_WUNDEF -- Performing Test HAVE_WUNDEF - Success -- Performing Test HAVE_WVLA -- Performing Test HAVE_WVLA - Success -- Looking for clock_gettime -- Looking for clock_gettime - found -- Looking for futimens -- Looking for futimens - found -- Looking for posix_fadvise -- Looking for posix_fadvise - found -- Looking for posix_madvise -- Looking for posix_madvise - found -- Performing Test HAVE_STAT_NANOSECOND_PRECISION -- Performing Test HAVE_STAT_NANOSECOND_PRECISION - Success -- Configuring done (1.6s) -- Generating done (0.0s) Then I ran make, which leads to the following errors: [ 3%] Building C object CMakeFiles/libdeflate_static.dir/lib/arm/cpu_features.c.o [ 6%] Building C object CMakeFiles/libdeflate_static.dir/lib/utils.c.o [ 10%] Building C object CMakeFiles/libdeflate_static.dir/lib/x86/cpu_features.c.o [ 13%] Building C object CMakeFiles/libdeflate_static.dir/lib/deflate_compress.c.o [ 17%] Building C object CMakeFiles/libdeflate_static.dir/lib/deflate_decompress.c.o [ 20%] Building C object CMakeFiles/libdeflate_static.dir/lib/adler32.c.o /tmp/cc00W0eO.s: Assembler messages: /tmp/cc00W0eO.s:644: Error: no such instruction:vpdpbusd %ymm17,%ymm2,%ymm6' /tmp/cc00W0eO.s:645: Error: no such instruction: vpdpbusd %ymm15,%ymm7,%ymm3' /tmp/cc00W0eO.s:647: Error: no such instruction:vpdpbusd %ymm14,%ymm2,%ymm6' /tmp/cc00W0eO.s:649: Error: no such instruction: vpdpbusd %ymm15,%ymm7,%ymm3' /tmp/cc00W0eO.s:650: Error: no such instruction:vpdpbusd %ymm14,%ymm2,%ymm6' /tmp/cc00W0eO.s:667: Error: no such instruction: vpdpbusd %ymm18,%ymm7,%ymm3' /tmp/cc00W0eO.s:668: Error: no such instruction:vpdpbusd %ymm1,%ymm7,%ymm6' /tmp/cc00W0eO.s:679: Error: no such instruction: vpdpbusd %ymm2,%ymm0,%ymm3' /tmp/cc00W0eO.s:680: Error: no such instruction:vpdpbusd %ymm1,%ymm0,%ymm6' /tmp/cc00W0eO.s:751: Error: no such instruction: vpdpbusd %ymm13,%ymm24,%ymm2' /tmp/cc00W0eO.s:752: Error: no such instruction:vpdpbusd %ymm13,%ymm23,%ymm1' /tmp/cc00W0eO.s:753: Error: no such instruction: vpdpbusd %ymm13,%ymm22,%ymm0' /tmp/cc00W0eO.s:754: Error: no such instruction:vpdpbusd %ymm13,%ymm21,%ymm11' /tmp/cc00W0eO.s:755: Error: no such instruction: vpdpbusd %ymm12,%ymm24,%ymm6' /tmp/cc00W0eO.s:756: Error: no such instruction:vpdpbusd %ymm12,%ymm23,%ymm5' /tmp/cc00W0eO.s:757: Error: no such instruction: vpdpbusd %ymm12,%ymm22,%ymm4' /tmp/cc00W0eO.s:758: Error: no such instruction:vpdpbusd %ymm12,%ymm21,%ymm3' /tmp/cc00W0eO.s:850: Error: no such instruction: vpdpbusd %zmm17,%zmm2,%zmm6' /tmp/cc00W0eO.s:851: Error: no such instruction:vpdpbusd %zmm15,%zmm7,%zmm3' /tmp/cc00W0eO.s:853: Error: no such instruction: vpdpbusd %zmm14,%zmm2,%zmm6' /tmp/cc00W0eO.s:855: Error: no such instruction:vpdpbusd %zmm15,%zmm7,%zmm3' /tmp/cc00W0eO.s:856: Error: no such instruction: vpdpbusd %zmm14,%zmm2,%zmm6' /tmp/cc00W0eO.s:873: Error: no such instruction:vpdpbusd %zmm18,%zmm7,%zmm3' /tmp/cc00W0eO.s:874: Error: no such instruction: vpdpbusd %zmm1,%zmm7,%zmm6' /tmp/cc00W0eO.s:885: Error: no such instruction:vpdpbusd %zmm2,%zmm0,%zmm3' /tmp/cc00W0eO.s:886: Error: no such instruction: vpdpbusd %zmm1,%zmm0,%zmm6' /tmp/cc00W0eO.s:961: Error: no such instruction:vpdpbusd %zmm13,%zmm24,%zmm2' /tmp/cc00W0eO.s:962: Error: no such instruction: vpdpbusd %zmm13,%zmm23,%zmm1' /tmp/cc00W0eO.s:963: Error: no such instruction:vpdpbusd %zmm13,%zmm22,%zmm0' /tmp/cc00W0eO.s:964: Error: no such instruction: vpdpbusd %zmm13,%zmm21,%zmm11' /tmp/cc00W0eO.s:965: Error: no such instruction:vpdpbusd %zmm12,%zmm24,%zmm6' /tmp/cc00W0eO.s:966: Error: no such instruction: vpdpbusd %zmm12,%zmm23,%zmm5' /tmp/cc00W0eO.s:967: Error: no such instruction:vpdpbusd %zmm12,%zmm22,%zmm4' /tmp/cc00W0eO.s:968: Error: no such instruction: vpdpbusd %zmm12,%zmm21,%zmm3' make[2]: *** [CMakeFiles/libdeflate_static.dir/lib/adler32.c.o] Error 1 make[1]: *** [CMakeFiles/libdeflate_static.dir/all] Error 2 make: *** [all] Error 2 /tmp/cc00W0eO.s: Assembler messages: /tmp/cc00W0eO.s:644: Error: no such instruction: `vpdpbusd %ymm17,%ymm2,%ymm6' ... (additional errors) Despite the flags -mno-avx, -mno-avx2, and -mno-avx512f, the assembler output shows attempts to use vpdpbusd instructions, which should have been disabled. My machine supports SSE, AVX, and AVX2 instructions but does not support AVX-512.

ebiggers commented 3 weeks ago

This is working as intended. libdeflate expects that your binutils version is not significantly older than your gcc version. CentOS 7 comes with gcc 4.8, so it sounds like you upgraded to gcc 9.3 without upgrading binutils alongside it. You need to upgrade binutils too in order to have a complete upgraded toolchain. If you are using a common combination that I overlooked, e.g. the gcc and binutils that were shipped with a particular version of a distro, I can increase the gcc version that libdeflate checks for to cover that. But I can't work around arbitrary custom gcc + binutils combinations, considering that libdeflate's source code is buildable without a configuration step.

-no-avx512f only means "do not generate AVX512 code that isn't guarded by a runtime CPU feature check". libdeflate has optimized code guarded by runtime CPU feature checks, so that is why AVX512 code still appears in the binary. Note also that -no-avx512f is already the default. Likewise for -no-avx and -no-avx2.

ebiggers commented 1 week ago

I am thinking about submitting a workaround that would update the CMake-based build system to disable some optimized code in cases where the binutils is too old. Can you please try https://github.com/ebiggers/libdeflate/pull/389 and check that it allows libdeflate to build on your system with the toolchain you were trying to use? (Upgrading binutils alongside gcc is still the proper solution to the problem, of course.)

ebiggers commented 1 week ago

https://github.com/ebiggers/libdeflate/pull/389 has been submitted.