search

Legrandin / pycryptodome

A self-contained cryptographic library for Python
https://www.pycryptodome.org
Other
2.74k stars 492 forks source link

Build failure with GCC 14 and LTO if -maes is not explicitly enabled #787

Closed kostadinsh closed 6 months ago

kostadinsh commented 6 months ago

pycryptodome fails to build with GCC 14 and LTO if -maes is not explicitly enabled

The build fails ONLY on GCC 14, GCC 13 is fine, if -maes is added to the CFLAGS or -flto is omitted, the build finishes fine with GCC 14. GCC14 is not yet released and won't be for months yet, i'm filing this to see first if pycryptodome's detection of CPU features is going wrong, or if you think I should report it to GCC instead.

Something that I noticed is that these two appear in the GCC 13 log, but not in the GCC 14 one:

Testing support for AESNI intrinsics
Target does support AESNI intrinsics

I have attached build logs from both GCC 13 and 14.

Steps to reproduce:

  1. have a working GCC 14 compiler and use that to build pycryptodome
  2. run git clone https://github.com/Legrandin/pycryptodome
  3. run export CFLAGS="-O2 -flto"
  4. run python3 setup.py sdist bdist_wheel
  5. build failure 
    In file included from src/AESNI.c:34:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h: In function ‘sub_rot’:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:71:18: note: called from here
71 |     case 1:  y = _mm_aeskeygenassist_si128(x, 0x01); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:72:18: note: called from here
72 |     case 2:  y = _mm_aeskeygenassist_si128(x, 0x02); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:73:18: note: called from here
73 |     case 3:  y = _mm_aeskeygenassist_si128(x, 0x04); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:74:18: note: called from here
74 |     case 4:  y = _mm_aeskeygenassist_si128(x, 0x08); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:75:18: note: called from here
75 |     case 5:  y = _mm_aeskeygenassist_si128(x, 0x10); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:76:18: note: called from here
76 |     case 6:  y = _mm_aeskeygenassist_si128(x, 0x20); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:77:18: note: called from here
77 |     case 7:  y = _mm_aeskeygenassist_si128(x, 0x40); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:78:18: note: called from here
78 |     case 8:  y = _mm_aeskeygenassist_si128(x, 0x80); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:79:18: note: called from here
79 |     case 9:  y = _mm_aeskeygenassist_si128(x, 0x1b); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:74:1: error: inlining failed in call to ‘always_inline’ ‘_mm_aeskeygenassist_si128’: target specific option mismatch
74 | _mm_aeskeygenassist_si128 (__m128i __X, const int __C)
  | ^~~~~~~~~~~~~~~~~~~~~~~~~
src/AESNI.c:80:18: note: called from here
80 |     case 10: y = _mm_aeskeygenassist_si128(x, 0x36); break;
  |                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command '/usr/bin/x86_64-pc-linux-gnu-gcc' failed with exit code 1

build log from GCC 13 build log from GCC 14

Legrandin commented 6 months ago

Thanks for the flagging. Can you suggest a way to replicate this problem without having to fully compile from scratch the most recent gcc14 candidate (e.g., some Docker image)?

kostadinsh commented 6 months ago

I don't think there are docker images for gcc itself, but you could use a gentoo image, and build gcc-14 with portage.

thesamesam commented 6 months ago

Let me see. I think there might be something I can arrange. I will get back to you.

thesamesam commented 6 months ago

OK, try https://jwakely.github.io/pkg-gcc-latest/. They're unofficial builds from one of the GCC maintainers.

The example Dockerfile on that page works for me:

FROM ubuntu:latest
RUN apt-get update -y
RUN apt-get install -y build-essential wget
RUN wget --quiet https://kayari.org/gcc-latest/gcc-latest.deb
RUN dpkg -i gcc-latest.deb
RUN /opt/gcc-latest/bin/gcc --version

with:

docker build -t gcc gcc/
docker run -it gcc
Legrandin commented 6 months ago

Thanks @thesamesam! Unfortunately, with that GCC14 version and OS, I can't reproduce the problem (as in, the library builds fine with export CFLAGS="-O2 -lto".

@kostadinsh if this is still a problem for you, can you also retry by setting PYCRYPTODOME_DEBUG=1, then you get a complete dump of all the autodetect logic.

kostadinsh commented 6 months ago

These appeard with the env variable set:

build/test1.c:3:6: error: #error Not clang
    3 |     #error Not clang
      |      ^~~~~
build/test1.c:2:14: fatal error: intrin.h: No such file or directory
    2 |     #include <intrin.h>
      |              ^~~~~~~~~~
compilation terminated.
build/test1.c:2:14: fatal error: intrin.h: No such file or directory
    2 |     #include <intrin.h>
      |              ^~~~~~~~~~
compilation terminated.
In file included from build/test1.c:2:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h: In function ‘f’:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:103:1: error: inlining failed in call to ‘always_inline’ ‘_mm_clmulepi64_si128’: target specific option mismatch
  103 | _mm_clmulepi64_si128 (__m128i __X, __m128i __Y, const int __I)
      | ^~~~~~~~~~~~~~~~~~~~
build/test1.c:6:16: note: called from here
    6 |         return _mm_clmulepi64_si128(x, y, 0x00);
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

complete build log

Legrandin commented 6 months ago

I think I found the issue, and pushed a fix to master (b4083688fde0580de6c2a4d36d84da31a2549a2c). Please try if you have a chance.

kostadinsh commented 6 months ago

A few new things popped up with the debug variable, but the build finishes fine.

build/test1.c:3:6: error: #error Not clang
    3 |     #error Not clang
      |      ^~~~~
build/test1.c:2:14: fatal error: intrin.h: No such file or directory
    2 |     #include <intrin.h>
      |              ^~~~~~~~~~
compilation terminated.
build/test1.c:2:14: fatal error: intrin.h: No such file or directory
    2 |     #include <intrin.h>
      |              ^~~~~~~~~~
compilation terminated.
In function ‘f’,
    inlined from ‘main’ at build/test1.c:11:13:
build/test1.c:5:16: error: ‘__builtin_ia32_aesenc128’ needs isa option -mvaes -mavx512vl -maes -msse2
    5 |         return _mm_aesenc_si128(x, y);
      |                ^
lto-wrapper: fatal error: /usr/bin/x86_64-pc-linux-gnu-gcc returned 1 exit status
compilation terminated.
/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status
In file included from build/test1.c:2:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h: In function ‘f’:
/usr/lib/gcc/x86_64-pc-linux-gnu/14/include/wmmintrin.h:103:1: error: inlining failed in call to ‘always_inline’ ‘_mm_clmulepi64_si128’: target specific option mismatch
  103 | _mm_clmulepi64_si128 (__m128i __X, __m128i __Y, const int __I)
      | ^~~~~~~~~~~~~~~~~~~~
build/test1.c:6:16: note: called from here
    6 |         return _mm_clmulepi64_si128(x, y, 0x00);
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

complete build log

Legrandin commented 6 months ago

Yes, when debug is active, those error messages are expected (as part of the trial-and-error to autodetect).

This bug is fixed in v3.20.0.

thesamesam commented 6 months ago

Just want to say thank you for addressing this quickly and the assistance. It helps a lot when testing many packages with upcoming releases.