search

isce-framework / fringe

Fine Resolution InSAR With Generalized Eigenvectors (FRInGE)
Apache License 2.0
80 stars 42 forks source link

MacOS: Segmentation Fault 11 When Importing C++ Modules #42

Open rbiessel opened 3 years ago

rbiessel commented 3 years ago

Hi All,

I'm getting a segmentation fault whenever the c++ modules are imported.

(fringe) MBP:fringe rbiessel$ python
Python 3.7.7 (default, Mar 23 2020, 17:31:31) 
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmaplib
Segmentation fault: 11

The same thing happens when running nmap.py. Has anyone else encountered this? Is this likely an installation problem? I've tried reinstalling the package without any luck.

Here's the build/installation process I followed according to the installation instructions:

cmake ``` :build rbiessel$ CXX=${CXX} cmake -DCMAKE_INSTALL_PREFIX=../install ../src/fringe -DCMAKE_FIND_FRAMEWORK=NEVER -- The C compiler identification is Clang 10.0.0 -- The CXX compiler identification is Clang 10.0.0 -- Check for working C compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang -- Check for working C compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang++ -- Check for working CXX compiler: /opt/anaconda3/envs/fringe/bin/x86_64-apple-darwin13.4.0-clang++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found GDAL: /opt/anaconda3/envs/fringe/lib/libgdal.dylib (found suitable version "3.0.2", minimum required is "3.0") -- Found Armadillo: /opt/local/lib/libarmadillo.dylib (found suitable version "10.3.0", minimum required is "8.100") -- Found OpenMP_CXX: -fopenmp=libomp (found version "4.5") -- Found OpenMP: TRUE (found version "4.5") found components: CXX -- Looking for sgemm_ -- Looking for sgemm_ - not found -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - found -- Found Threads: TRUE -- Looking for sgemm_ -- Looking for sgemm_ - found -- Found BLAS: /opt/anaconda3/envs/fringe/lib/libmkl_intel_lp64.dylib;/opt/anaconda3/envs/fringe/lib/libmkl_intel_thread.dylib;/opt/anaconda3/envs/fringe/lib/libmkl_core.dylib;/opt/anaconda3/envs/fringe/lib/libiomp5.dylib;-lm;-ldl -- Looking for cheev_ -- Looking for cheev_ - found -- A library with LAPACK API found. -- Found Python: /opt/anaconda3/envs/fringe/bin/python3.7 (found suitable version "3.7.7", minimum required is "3.7") found components: Interpreter Development NumPy -- Found Python: /opt/anaconda3/envs/fringe/bin/python3.7 (found suitable version "3.7.7", minimum required is "3.6") found components: Interpreter -- Found Cython: /opt/local/bin/cython-3.7 -- Found Python: /opt/anaconda3/envs/fringe/lib/libpython3.7m.dylib (found version "3.7.7") found components: Development -- Found Cython: 0.29.22 -- Configuring done -- Generating done -- Build files have been written to: /Users/rbiessel/Documents/apps/ISCE/fringe/build ```
make all ``` :build rbiessel$ make all [ 3%] Compiling Cython CXX source for calamplib... Scanning dependencies of target calamplib [ 7%] Building CXX object src/calamp/CMakeFiles/calamplib.dir/calamplib.cxx.o [ 10%] Linking CXX shared module calamplib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 10%] Built target calamplib [ 14%] Compiling Cython CXX source for ampdispersionlib... Scanning dependencies of target ampdispersionlib [ 17%] Building CXX object src/ampdispersion/CMakeFiles/ampdispersionlib.dir/ampdispersionlib.cxx.o [ 21%] Linking CXX shared module ampdispersionlib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 21%] Built target ampdispersionlib [ 25%] Compiling Cython CXX source for nmaplib... Scanning dependencies of target nmaplib [ 28%] Building CXX object src/nmap/CMakeFiles/nmaplib.dir/nmaplib.cxx.o [ 32%] Linking CXX shared module nmaplib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 32%] Built target nmaplib [ 35%] Compiling Cython CXX source for despecklib... Scanning dependencies of target despecklib [ 39%] Building CXX object src/despeck/CMakeFiles/despecklib.dir/despecklib.cxx.o [ 42%] Linking CXX shared module despecklib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 42%] Built target despecklib [ 46%] Compiling Cython CXX source for evdlib... Scanning dependencies of target evdlib [ 50%] Building CXX object src/evd/CMakeFiles/evdlib.dir/evdlib.cxx.o [ 53%] Linking CXX shared module evdlib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 53%] Built target evdlib [ 57%] Compiling Cython CXX source for sbaslib... Scanning dependencies of target sbaslib [ 60%] Building CXX object src/sbas/CMakeFiles/sbaslib.dir/sbaslib.cxx.o [ 64%] Linking CXX shared module sbaslib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 64%] Built target sbaslib Scanning dependencies of target fit [ 67%] Building CXX object src/fit/CMakeFiles/fit.dir/Fit.cc.o [ 71%] Linking CXX shared library libfit.dylib ld: warning: -pie being ignored. It is only used when linking a main executable [ 71%] Built target fit [ 75%] Compiling Cython CXX source for fitlib... Scanning dependencies of target fitlib [ 78%] Building CXX object src/fit/CMakeFiles/fitlib.dir/fitlib.cxx.o [ 82%] Linking CXX shared module fitlib.so ld: warning: -pie being ignored. It is only used when linking a main executable [ 82%] Built target fitlib Scanning dependencies of target unwrap [ 85%] Building CXX object src/unwrap_error/CMakeFiles/unwrap.dir/Unwrap.cc.o [ 89%] Linking CXX shared library libunwrap.dylib ld: warning: -pie being ignored. It is only used when linking a main executable [ 89%] Built target unwrap [ 92%] Compiling Cython CXX source for unwraplib... Scanning dependencies of target unwraplib [ 96%] Building CXX object src/unwrap_error/CMakeFiles/unwraplib.dir/unwraplib.cxx.o [100%] Linking CXX shared module unwraplib.so ld: warning: -pie being ignored. It is only used when linking a main executable [100%] Built target unwraplib ```
make install ``` :build rbiessel$ make install [ 10%] Built target calamplib [ 21%] Built target ampdispersionlib [ 32%] Built target nmaplib [ 42%] Built target despecklib [ 53%] Built target evdlib [ 64%] Built target sbaslib [ 71%] Built target fit [ 82%] Built target fitlib [ 89%] Built target unwrap [100%] Built target unwraplib Install the project... -- Install configuration: "" -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/calamplib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/calamp.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/ampdispersionlib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/ampdispersion.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/nmap.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/despecklib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/despeck.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/evdlib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/evd.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/Stack.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/sequential.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/sbaslib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/sbas.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/lib/libfit.dylib -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/fitlib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/TimeFunction.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/fit_timeseries.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/BasisFn.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/CenteredBasisFn.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Collection.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Constant.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Exp.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Interface.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Log.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Periodic.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Polynomial.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Power.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Spline.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/SplineFactory.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/Trigonometry.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/__init__.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/timefn/datetimeUtils.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/lib/libunwrap.dylib -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/unwraplib.so -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/uavsar2vrt.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/stripmap2vrt.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/tops2vrt.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/adjustMiniStacks.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/unwrap_fringe.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/unwrapStack.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/Network.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/StackVRT.py -- Installing: /Users/rbiessel/Documents/apps/ISCE/fringe/install/bin/integratePS.py ```
rbiessel commented 3 years ago

Hi, @hfattahi / @piyushrpt, I spent some more time trying to get around this issue and I was wondering if either of you had any insight? I'm also hoping to contribute code eventually. Do either of you have any suggested resources pertaining development workflows via MacOS?

piyushrpt commented 3 years ago

new macs are a little weird. gcc is pretty much unusable. Use clang. CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX={installdir} {srcdir}

rbiessel commented 3 years ago

new macs are a little weird. gcc is pretty much unusable. Use clang. CC=clang CXX=clang++ cmake -DCMAKE_INSTALL_PREFIX={installdir} {srcdir}

I've been using clang to build FRInGE from the start but I have tried gcc as well. Clang does seem to get farther in the process but I'm still getting a segmentation fault. This time with Clang 11.0.0:

Python 3.7.8 | packaged by conda-forge | (default, Nov 27 2020, 19:11:24) 
[Clang 11.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmaplib
Segmentation fault: 11

Do you think this ld: warning is the issue?

[ 25%] Compiling Cython CXX source for nmaplib...
Scanning dependencies of target nmaplib
[ 28%] Building CXX object src/nmap/CMakeFiles/nmaplib.dir/nmaplib.cxx.o
[ 32%] Linking CXX shared module nmaplib.so
ld: warning: -pie being ignored. It is only used when linking a main executable
[ 32%] Built target nmaplib
piyushrpt commented 3 years ago

clang-11 and gcc-10 are both a little weird as they get stricter with what can be done. I have only tested cmake with clang 9/10. Maybe this will help - https://andreasfertig.blog/2021/02/clang-and-gcc-on-macos-catalina-finding-the-include-paths/

rbiessel commented 3 years ago

Building with clang 10 (the mp part seemed necessary, otherwise cmake couldn't find OpenMP)

rbiessel$ CXX=clang++-mp-10 CC=clang-mp-10 cmake -DCMAKE_FIND_FRAMEWORK=NEVER -DCMAKE_INSTALL_PREFIX=../install ../src/fringe
Python 3.7.8 | packaged by conda-forge | (default, Nov 27 2020, 19:11:24) 
[Clang 11.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import nmaplib
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so, 2): Symbol not found: _PyObject_GC_IsFinalized
  Referenced from: /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so
  Expected in: flat namespace
 in /Users/rbiessel/Documents/apps/ISCE/fringe/install/python/nmaplib.so
piyushrpt commented 3 years ago

Can you post the output of cmake - this looks like the right python is not being used / discovered

rbiessel commented 3 years ago

I was reading this kind of error could be caused by packages compiled with conflicting libraries?

Might it be because it's finding packages outside of the conda environment? 

(fringe) Rowans-MBP:build rbiessel$ CXX=clang++-mp-10 CC=clang-mp-10 cmake -DCMAKE_PREFIX_PATH=$CONDA_PREFIX  -DCMAKE_INSTALL_PREFIX=../install ../src/fringe
-- The C compiler identification is Clang 10.0.1
-- The CXX compiler identification is Clang 10.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/local/bin/clang-mp-10 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/local/bin/clang++-mp-10 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found GDAL: /opt/anaconda3/envs/fringe/lib/libgdal.dylib (found suitable version "3.0.4", minimum required is "3.0") 
-- Found Armadillo: /opt/anaconda3/envs/fringe/include (found suitable version "9.900.4", minimum required is "8.100") 
-- Found OpenMP_CXX: -fopenmp=libomp (found version "4.5") 
-- Found OpenMP: TRUE (found version "4.5") found components: CXX 
-- Looking for sgemm_
-- Looking for sgemm_ - not found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Looking for sgemm_
-- Looking for sgemm_ - found
-- Found BLAS: /opt/local/lib/libopenblas.dylib  
-- Looking for cheev_
-- Looking for cheev_ - found
-- Found LAPACK: /opt/local/lib/libopenblas.dylib;-lm;-ldl  
-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found suitable version "3.9.5", minimum required is "3.7") found components: Interpreter Development NumPy Development.Module Development.Embed 
-- Found Python: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found suitable version "3.9.5", minimum required is "3.6") found components: Interpreter 
-- Found Cython: /opt/local/bin/cython-3.9  
-- Found Python: /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 (found version "3.9.5") found components: Development Development.Module Development.Embed 
-- Found Cython:  0.29.22
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/rbiessel/Documents/apps/ISCE/fringe/build
(fringe) Row
rbiessel commented 3 years ago

Actually, I got nmaplib.py (and other modules) to import succesfully using python3.9 (instead of the conda enviromnent's python3.7)! I will now see if I can use the workflow and if it will actually work! This is the farthest I've gotten with it so far on this macbook though. Thank you for your help Piyush

piyushrpt commented 3 years ago

Use -DPython3_EXECUTABLE=path-to-python3-you-want during cmake to make it specifically use that version. Or use -DCMAKE_FIND_FRAMEWORK=NEVER to use python3 that is active in the environment.