Open TheChymera opened 7 months ago
I had forgotten to put the link for the full build and test log, it's now here as well as in the corrected link in the original post.
Ok, so apparently for some reason the import fails if the package is built without the lto
CFLAG and with SSE2/AVX2 support:
This build (CFLAGS are "-march=native -O2 -pipe"), results in:
[deco]~ ❱ python -c "from numcodecs import lz4"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: /usr/lib/python3.11/site-packages/numcodecs/lz4.cpython-311-x86_64-linux-gnu.so: undefined symbol: LZ4_compressBound
This build (CFLAGS are "-march=native -O2 -pipe -flto=auto") works fine.
This build (CFLAGS are "-march=native -O2 -pipe" but it also has DISABLE_NUMCODECS_AVX2=1
and DISABLE_NUMCODECS_SSE2=1
) also works fine.
I have no idea what this could be about. Do any of you know more?
cc: @jakirkham
I was working out a similar issue earlier. From my experience that error occurred when the lz4.h was not found when the .pyx is compiled.
For me adding the following to the extension resolved the issue
libraries="lz4"
@leaver2000 what do you mean by “adding” — is this some sort of patch that could be applied to the source code to make it less fragile to compiler options?
@TheChymera The error you described was identical to the issue I ran into when the lz4_sources could not be found during setup
lz4.cpython-311-x86_64-linux-gnu.so: undefined symbol: LZ4_compressBound
https://github.com/zarr-developers/numcodecs/blob/main/setup.py#L188
extensions = [
Extension('numcodecs.lz4',
sources=sources + lz4_sources,
include_dirs=include_dirs,
define_macros=define_macros,
extra_compile_args=extra_compile_args,
libraries="lz4", # adding resolved my issues.
),
]
Maybe just chuck a in an assert into the setup.py, to insure your compiler flags are not messing the install.
assert lz4_sources != []
For context, though probably not relevant, I am trying to debug this issue with a package which uses numcodecs.
I have installed numcodecs via the Gentoo Linux package manager full build and test log here. The lz4 test is indeed skipped:
But the build says the module is being built:
Any idea what might be going on here?