search

imaris / ImarisWriterTest

Apache License 2.0
1 stars 3 forks source link

Segmentation fault with PyImarisWriter tests on CentOS 7 #3

Closed carshadi closed 2 years ago

carshadi commented 2 years ago

Hi all,

We are interested in using PyimarisWriter to compress terabyte-scale lightsheet datasets. I was able to compile ImarisWriter and all its dependencies on CentOS 7 using cmake 3.22.1 and gcc/g++ 9.3.1. I can run TestPyImarisWriter.py successfully, which gives the output:

Ran 17 tests in 0.019s

OK

However, if I run ImarisWriterCtypesTest.py I get a segfault. Running from gdb:

gdb python
run ImarisWriterCtypesTest.py

Outputs the following:

Starting program: /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/bin/python ImarisWriterCtypesTest.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/site-packages/numpy/core/../../../.././libgfortran.so.5]
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/site-packages/numpy/core/../../../../././libquadmath.so.0]
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/site-packages/numpy/core/../../../../././libgcc_s.so.1]
[New Thread 0x7fffee0b5700 (LWP 23737)]
[New Thread 0x7fffed8b4700 (LWP 23738)]
[New Thread 0x7fffe50b3700 (LWP 23739)]
[Thread 0x7fffed8b4700 (LWP 23738) exited]
[Thread 0x7fffe50b3700 (LWP 23739) exited]
[Thread 0x7fffee0b5700 (LWP 23737) exited]
[Detaching after fork from child process 23740]
Adding to PATH variable: /home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter
start 13:50:08
Loading dll: /home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter/libbpImarisWriter96.so
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/bin/../lib/libstdc++.so.6]
Writing out_0py.ims
[New Thread 0x7fffe50b3700 (LWP 23741)]
[New Thread 0x7fffed8b4700 (LWP 23742)]
[New Thread 0x7fffee0b5700 (LWP 23744)]
[New Thread 0x7fffd2f7a700 (LWP 23745)]
[New Thread 0x7fffd2779700 (LWP 23746)]
[New Thread 0x7fffd1f78700 (LWP 23747)]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff704f8c1 in __strlen_sse2_pminub () from /lib64/libc.so.6

The error occurs within this function https://github.com/imaris/ImarisWriterTest/blob/2de64e247964eec9e85c89b4a39c9852c046d1e0/testPy/ImarisWriterCtypesTest.py#L69

A segfault also occurs with PyImarisWriterExample.py 

Starting program: /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/bin/python PyImarisWriterExample.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/site-packages/numpy/core/../../../.././libgfortran.so.5]
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/site-packages/numpy/core/../../../../././libquadmath.so.0]
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/site-packages/numpy/core/../../../../././libgcc_s.so.1]
[New Thread 0x7fffee0b5700 (LWP 24535)]
[New Thread 0x7fffed8b4700 (LWP 24536)]
[New Thread 0x7fffdd0b3700 (LWP 24537)]
[Thread 0x7fffee0b5700 (LWP 24535) exited]
[Thread 0x7fffdd0b3700 (LWP 24537) exited]
[Thread 0x7fffed8b4700 (LWP 24536) exited]
[Detaching after fork from child process 24538]
Adding to PATH variable: /home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter
Dwarf Error: wrong version in compilation unit header (is 5, should be 2, 3, or 4) [in module /home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/bin/../lib/libstdc++.so.6]

Program received signal SIGSEGV, Segmentation fault.
0x00007fffd3b44fa9 in Convert(bpConverterTypesC_Index5D const*) () from /home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter/libbpImarisWriter96.so

In this case the segfault is triggered by converter.NeedCopyBlock https://github.com/imaris/ImarisWriterTest/blob/2de64e247964eec9e85c89b4a39c9852c046d1e0/testPy/PyImarisWriterExample.py#L74

Here is what my PyImarisWriter package directory looks like, including the compiled library

ImarisWriterCtypes.py  
__init__.py 
libbpImarisWriter96.so  
__pycache__  
PyImarisWriter.py

Output of lscpu:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             4
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 61
Model name:            Intel Core Processor (Broadwell)
Stepping:              2
CPU MHz:               2793.436
BogoMIPS:              5586.87
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              4096K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm 3dnowprefetch invpcid_single fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt

cmake output:

CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Warning (dev) at CMakeLists.txt:3 (set):
  implicitly converting 'TYPE' to 'STRING' type.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- The C compiler identification is GNU 9.3.1
-- The CXX compiler identification is GNU 9.3.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-9/root/usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-9/root/usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- define an empty cmake module path
-- Found HDF5: /home/cameron.arshadi/lib/libhdf5.so;/home/cameron.arshadi/lib/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so (found version "1.12.1")
Found HDF.+/home/cameron.arshadi/include+  +/home/cameron.arshadi/lib/libhdf5.so/home/cameron.arshadi/lib/libz.so/usr/lib64/libdl.so/usr/lib64/libm.so
-- Found ZLIB: /home/cameron.arshadi/lib/libz.so (found version "1.2.11")
Found ZLIB.+/home/cameron.arshadi/include+  +/home/cameron.arshadi/lib/libz.so
-- Looking for LZ4 library below: /home/cameron.arshadi
-- Looking for LZ4 header files below: /home/cameron.arshadi
-- Found LZ4: /home/cameron.arshadi/lib/liblz4.so (found version "1.9.3")
Found LZ4.+/home/cameron.arshadi/include+  +/home/cameron.arshadi/lib/liblz4.so
Found build.+/home/cameron.arshadi/Downloads/ImarisWriter/release
-- Configuring done
-- Generating done
-- Build files have been written to: /home/cameron.arshadi/Downloads/ImarisWriter/release

make output:

[  1%] Building CXX object CMakeFiles/ImarisWriter_static.dir/c/bpImageConverterC.cxx.o
[  3%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpCompressionAlgorithmFactory.cxx.o
[  5%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpDeriche.cxx.o
[  7%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpGzip.cxx.o
[  9%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpH5LZ4.cxx.o
[ 11%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpHistogram.cxx.o
[ 12%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImageConverter.cxx.o
[ 14%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImageConverterImpl.cxx.o
[ 16%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImsImage3D.cxx.o
[ 18%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImsImage5D.cxx.o
[ 20%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImsImageBlock.cxx.o
[ 22%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImsLayout.cxx.o
[ 24%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImsLayout3D.cxx.o
[ 25%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpImsUtils.cxx.o
[ 27%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpLZ4.cxx.o
[ 29%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpMemoryManager.cxx.o
[ 31%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpMultiresolutionImsImage.cxx.o
[ 33%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpOptimalBlockLayout.cxx.o
[ 35%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpShuffle.cxx.o
[ 37%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpThreadPool.cxx.o
[ 38%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpThumbnailBuilder.cxx.o
[ 40%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpWriterCompressor.cxx.o
[ 42%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpWriterFactoryCompressor.cxx.o
[ 44%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpWriterFactoryHDF5.cxx.o
[ 46%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpWriterHDF5.cxx.o
[ 48%] Building CXX object CMakeFiles/ImarisWriter_static.dir/writer/bpWriterThreads.cxx.o
[ 50%] Linking CXX static library libImarisWriter_static.a
[ 50%] Built target ImarisWriter_static
[ 51%] Building CXX object CMakeFiles/bpImarisWriter96.dir/c/bpImageConverterC.cxx.o
[ 53%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpCompressionAlgorithmFactory.cxx.o
[ 55%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpDeriche.cxx.o
[ 57%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpGzip.cxx.o
[ 59%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpH5LZ4.cxx.o
[ 61%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpHistogram.cxx.o
[ 62%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImageConverter.cxx.o
[ 64%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImageConverterImpl.cxx.o
[ 66%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImsImage3D.cxx.o
[ 68%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImsImage5D.cxx.o
[ 70%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImsImageBlock.cxx.o
[ 72%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImsLayout.cxx.o
[ 74%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImsLayout3D.cxx.o
[ 75%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpImsUtils.cxx.o
[ 77%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpLZ4.cxx.o
[ 79%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpMemoryManager.cxx.o
[ 81%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpMultiresolutionImsImage.cxx.o
[ 83%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpOptimalBlockLayout.cxx.o
[ 85%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpShuffle.cxx.o
[ 87%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpThreadPool.cxx.o
[ 88%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpThumbnailBuilder.cxx.o
[ 90%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpWriterCompressor.cxx.o
[ 92%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpWriterFactoryCompressor.cxx.o
[ 94%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpWriterFactoryHDF5.cxx.o
[ 96%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpWriterHDF5.cxx.o
[ 98%] Building CXX object CMakeFiles/bpImarisWriter96.dir/writer/bpWriterThreads.cxx.o
[100%] Linking CXX shared library libbpImarisWriter96.so
[100%] Built target bpImarisWriter96

I then copied libbpImarisWriter96.so to the PyImarisWriter package folder before running the tests

Thank you for your help!

SachaGuyer commented 2 years ago

Hi,

There were some mistakes in the PyImarisWriter.py code that caused the segmentation faults on mac/linux. Interestingly, on windows it worked.

I have fixed the issues with these commits:

The same issue was also present in the ImarisWriterCtypesTest.py program, which I fixed in https://github.com/SachaGuyer/ImarisWriterTest/commit/a45564bfc888bafeb7b2141091164b36f3cebed2

With these changes I can now run all python test programs on macOS successfully. I have not tested centOS but assume that it should be fine. Let me know if it worked for you.

carshadi commented 2 years ago

Excellent, I will test this out and report back. Thank you!

carshadi commented 2 years ago

Hi @SachaGuyer , confirming this is working on my end.

The only thing was I got this error 

Traceback (most recent call last):
  File "/home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter/PyImarisWriter.py", line 353, in _load_dll
    self.mcdll = CDLL(dll_filename)
  File "/home/cameron.arshadi/miniconda3/envs/lightsheet-compression-tests/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libbpImarisWriter96.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "compress_h5.py", line 188, in <module>
    main()
  File "compress_h5.py", line 171, in main
    bytes_written, compress_time, cpu_utilization = run(test_config, np_data.astype(test_config.mNp_type), nCores)
  File "compress_h5.py", line 90, in run
    converter = PW.ImageConverter(configuration.mImaris_type, image_size, sample_size, dimension_sequence, block_size,
  File "/home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter/PyImarisWriter.py", line 326, in __init__
    self._load_dll()
  File "/home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter/PyImarisWriter.py", line 357, in _load_dll
    self.raise_creating_clex(text)
  File "/home/cameron.arshadi/repos/lightsheet-compression-tests/PyImarisWriter/PyImarisWriter.py", line 90, in raise_creating_clex
    raise PyImarisWriterException('Error creating {}: {}'.format(self.__class__.__name__, message))
PyImarisWriter.PyImarisWriter.PyImarisWriterException: Error creating ImageConverter: Could not load library "libbpImarisWriter96.so" (with dependencies hdf5 and zlib)
Please make sure the library is available and in the PATH environment variable

But that was fixed by changing the following https://github.com/carshadi/ImarisWriter/commit/5bc6b95e679772a6f41694440892a067f3203cb9

SachaGuyer commented 2 years ago

Hi, I'm glad to hear that it now works on your system!

Thanks for the fix for loading the shared object. Apparently the behaviour of ctypes is slightly different on windows and mac/linux. On windows it is enough to add the folder where the shared object is located to the path, like it is done in PyImarisWriter. On mac/linux, the full path to the shared object file must be specified.

carshadi commented 2 years ago

I'll go ahead and close this