test failure after pip install

[JonLoveday]

I just did pip install Corrfunc without problems. However, the tests fail:

(base) loveday@mps024704 ~ % ipython
Python 3.9.12 (main, Apr  5 2022, 01:53:17) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from Corrfunc.tests import tests

In [2]: tests()
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 tests()

File ~/opt/anaconda3/lib/python3.9/site-packages/Corrfunc/tests.py:23, in tests()
     14 """
     15 Wrapper to run the two scripts that should have been installed
     16 with the Corrfunc package.
   (...)
     19 run successfully, then the package is working correctly.
     20 """
     22 # Import the script for calling the theory extensions
---> 23 from Corrfunc import call_correlation_functions as ct
     24 # Import the script for calling the mocks extensions
     25 from Corrfunc import call_correlation_functions_mocks as cm

File ~/opt/anaconda3/lib/python3.9/site-packages/Corrfunc/call_correlation_functions.py:20, in <module>
     18 import Corrfunc
     19 from Corrfunc.io import read_catalog
---> 20 from Corrfunc._countpairs import\
     21     countpairs as DD_extn,\
     22     countpairs_rp_pi as DDrppi_extn,\
     23     countpairs_wp as wp_extn,\
     24     countpairs_xi as xi_extn,\
     25     countspheres_vpf as vpf_extn,\
     26     countpairs_s_mu as DDsmu_extn
     29 def main():
     30     tstart = time.time()

ImportError: dlopen(/Users/loveday/opt/anaconda3/lib/python3.9/site-packages/Corrfunc/_countpairs.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace (_gsl_rng_alloc)

In [3]: 

[lgarrison]

Thanks for the report! Just to confirm, it looks like this is on a Mac? Can you provide your operating system version?

Usually a shared library issue like this happens when Corrfunc is installed in an environment where a library is available (GSL, in this case) and is then executed in a different environment where that library is not available. So you might want to double-check that you are installing and running in the same shell environment, same modules (if applicable), same conda env, etc.

On the other hand, it looks like we're inserting the GSL path via rpath here, so I would have expected this to be pretty robust.

@manodeep might have to lend his Mac expertise here!

(I added some formatting to your comment for readability.)

[JonLoveday]

MacOS 12.5

I think I've only got one conda environment (a fresh install on a new machine), and a new install of gsl 2.7 (make installcheck gave no errors).

[lgarrison]

Corrfunc finds GSL via the gsl-config executable e.g. here. If you use gsl-config --prefix, does it show the GSL that you installed?

[JonLoveday]

Seems OK:

(base) loveday@mps024704 gsl-2.7 % gsl-config --prefix
/usr/local
(base) loveday@mps024704 gsl-2.7 % ls /usr/local
bin         include     munki       sbin
etc         lib         munkireport share
(base) loveday@mps024704 gsl-2.7 % ls /usr/local/lib
libgsl.25.dylib     libgsl.la           libgslcblas.dylib
libgsl.a            libgslcblas.0.dylib libgslcblas.la
libgsl.dylib        libgslcblas.a       pkgconfig
(base) loveday@mps024704 gsl-2.7 %lsa /usr/local/lib
total 35592
drwxr-xr-x  11 root  wheel       352 26 Aug 15:15 .
drwxr-xr-x  10 root  wheel       320 26 Aug 15:15 ..
-rwxr-xr-x   1 root  wheel   3225164 26 Aug 15:15 libgsl.25.dylib
-rw-r--r--   1 root  wheel  13588280 26 Aug 15:15 libgsl.a
lrwxr-xr-x   1 root  wheel        15 26 Aug 15:15 libgsl.dylib -> libgsl.25.dylib
-rwxr-xr-x   1 root  wheel       910 26 Aug 15:15 libgsl.la
-rwxr-xr-x   1 root  wheel    266368 26 Aug 15:15 libgslcblas.0.dylib
-rw-r--r--   1 root  wheel   1124448 26 Aug 15:15 libgslcblas.a
lrwxr-xr-x   1 root  wheel        19 26 Aug 15:15 libgslcblas.dylib -> libgslcblas.0.dylib
-rwxr-xr-x   1 root  wheel       937 26 Aug 15:15 libgslcblas.la
drwxr-xr-x   3 root  wheel        96 26 Aug 15:15 pkgconfig
(base) loveday@mps024704 gsl-2.7 % 

[lgarrison]

I agree, looks okay. You might try installing from source, although I'm not sure why that would be different. You might also try export DYLD_LIBRARY_PATH=/usr/local/lib:$DYLD_LIBRARY_PATH (or Mac equivalent), although I would have thought that path would already be there.

@manodeep, can you take a look at this when you get the chance? A quick search brings up this (old) post which suggests the rpath syntax on MacOS is different than on Linux, could be relevant? https://stackoverflow.com/questions/4513799/how-to-set-the-runtime-path-rpath-of-an-executable-with-gcc-under-mac-osx

[manodeep]

@JonLoveday If you just want to fix the issue, then the fastest way would be to uninstall Corrfunc, install gsl through conda (conda install -c conda-forge gsl), and then re-install Corrfunc (python -m pip install Corrfunc).

If you want to debug what’s going, then I would first uninstall Corrfunc, and then install Corrfunc from source (ie git clone https://github.com/manodeep/Corrfunc) and then check if the regular install is working through - make && make tests. If that all works out (as expected), then please install the python wrappers with python -m pip —verbose install . &>install.log from the Corrfunc root directory (not sure how to enable verbose with the -m pip). The problem should still remain but at least we will be able to see what’s happening with the install paths.

[JonLoveday]

Many thanks Manodeep. Reinstalling gsl via conda did the trick, and was much faster than installing from source. Cheers, Jon

[manodeep]

Great - glad that the issue got sorted!