python / cpython

The Python programming language
https://www.python.org
Other
63.36k stars 30.34k forks source link

HACL SIMD support incorrectly detected under WASI #126458

Open brettcannon opened 2 hours ago

brettcannon commented 2 hours ago

Bug report

Bug description:

I'm getting a lot of:

/opt/wasi-sdk/bin/clang -c -I../../Modules/_hacl -I../../Modules/_hacl/include -D_BSD_SOURCE -D_DEFAULT_SOURCE -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -Wno-unused-value -Wno-empty-body -Qunused-arguments -Wno-parentheses-equality -Wno-unused-value -Wno-empty-body -Qunused-arguments -Wno-parentheses-equality  -std=c11 -Wextra -Wno-unused-parameter -Wno-int-conversion -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../../Include/internal -I../../Include/internal/mimalloc -IObjects -IInclude -IPython -I. -I../../Include     -msimd128 -msse -msse2 -msse3 -msse4.1 -msse4.2 -DHACL_CAN_COMPILE_VEC128 -o Modules/_hacl/Hacl_Hash_Blake2s_Simd128.o ../../Modules/_hacl/Hacl_Hash_Blake2s_Simd128.c
In file included from ../../Modules/_hacl/Hacl_Hash_Blake2s_Simd128.c:26:
In file included from ../../Modules/_hacl/internal/Hacl_Hash_Blake2s_Simd128.h:40:
../../Modules/_hacl/internal/../Hacl_Hash_Blake2s_Simd128.h:55:3: error: unknown type name 'Lib_IntVector_Intrinsics_vec128'

That seems to be defined at: https://github.com/python/cpython/blob/5e9168492f12c579b2481f3f3e0ae11f9d986857/Modules/_hacl/libintvector.h#L30

But that whole header file is guarded by: https://github.com/python/cpython/blob/5e9168492f12c579b2481f3f3e0ae11f9d986857/Modules/_hacl/libintvector.h#L22

Unfortunately, if you add defined(__wasi__) to that guard you then get:

/opt/wasi-sdk-24.0-x86_64-linux/lib/clang/18/include/immintrin.h:14:2: error: "This header is only meant to be used on x86 and x64 architecture"
   14 | #error "This header is only meant to be used on x86 and x64 architecture"
      |  ^

To me that suggests the check at: https://github.com/python/cpython/blob/5e9168492f12c579b2481f3f3e0ae11f9d986857/configure.ac#L7854-L7906

is too broad since the header files don't work on non-x86 architectures.

This probably requires either a CPU/platform check before checking the flag support or an explicit opt-out for at least WASI.

(And I have no idea why CI isn't running into this problem while I am locally; bad configure caching via https://github.com/python/cpython/blob/5e9168492f12c579b2481f3f3e0ae11f9d986857/.github/workflows/reusable-wasi.yml#L63-L68 ?)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Other

brettcannon commented 2 hours ago

@erlend-aasland how would you prefer to structure this in configure.ac?

erlend-aasland commented 2 hours ago

@erlend-aasland how would you prefer to structure this in configure.ac?

I think @picnixz already started hashing out better SIMD checks in #125022.