search

peterwittek / somoclu

Massively parallel self-organizing maps: accelerate training on multicore CPUs, GPUs, and clusters
https://peterwittek.github.io/somoclu/
MIT License
268 stars 70 forks source link

problems with python and somoclu on osx - no training, missing a linked library? #123

Closed jazoza closed 6 years ago

jazoza commented 6 years ago

Hello, I wanted to test the but have not been able to get it running. I have two machines: a 13'' MacBook (osx 10.13.4 no GPU) and a Mac Pro desktop (osx 10.10.5) with an NVIDIA GeForce GTX 650 Ti and CUDA Driver Version: 7.5.27. Laptop is running python 3, desktop 2.7, both through anaconda.

when installing somoclu with the standard command

conda install somoclu

I get this error when importing to python:

>>>import somoclu


RuntimeError: module compiled against API version 0xc but this version of numpy is 0xa
Warning: the binary library cannot be imported. You cannot train maps, but you can load and analyze ones that you have already saved.
If you installed Somoclu with pip on macOS, this typically means missing a linked library. If you compiled Somoclu with GCC, please make sure you have set DYLD_LIBRARY_PATH to include the GCC path. For more information, please refer to the documentation.```

When I install somoclu using the latest code from the git repository, as suggested in the documentation, the error is simply that there is a missing library and that I cannot train.

`>>> import somoclu`
```Warning: the binary library cannot be imported. You cannot train maps, but you can load and analyze ones that you have already saved.
If you installed Somoclu with pip on macOS, this typically means missing a linked library. If you compiled Somoclu with GCC, please make sure you have set DYLD_LIBRARY_PATH to include the GCC path. For more information, please refer to the documentation.```

This is the same on both machines.

On the desktop, I have installed CUDA drivers and toolkit, 

`$ /usr/local/cuda/bin/nvcc --version` gives the following output:

```nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Mon_Apr_11_13:23:40_CDT_2016
Cuda compilation tools, release 7.5, V7.5.26```

`$ gcc --version` is the following:

```Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.1.0 (clang-602.0.49) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.5.0
Thread model: posix```

I have set the CUDAHOME variable and the python install script is able to detect it.

During the `sudo python setup.py install` build, the nvcc command fails with the error:

`nvcc fatal   : The version ('60000') of the host compiler ('clang') is not supported`
Which, as far as I understand is caused by the incompatibility of my cuda and my clang versions. I downgraded xcode and clang from 7 to 6, but the issue persists. 

The install however, completes, and the result is the same: no training, missing library. 

`>>> import somoclu`
```Warning: the binary library cannot be imported. You cannot train maps, but you can load and analyze ones that you have already saved.
If you installed Somoclu with pip on macOS, this typically means missing a linked library. If you compiled Somoclu with GCC, please make sure you have set DYLD_LIBRARY_PATH to include the GCC path. For more information, please refer to the documentation```

Could you please tell me where in the documentation I can find more about which library to look for and whether this error is cause by a dependency conflict on (both of) my machines or is it rather a matter of OSX? How could I get somoclu to run on either on these? 

Thank you!

Here is the entire output of the install process: 

```running install
running bdist_egg
running egg_info
creating somoclu.egg-info
writing requirements to somoclu.egg-info/requires.txt
writing somoclu.egg-info/PKG-INFO
writing top-level names to somoclu.egg-info/top_level.txt
writing dependency_links to somoclu.egg-info/dependency_links.txt
writing manifest file 'somoclu.egg-info/SOURCES.txt'
reading manifest file 'somoclu.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'doc/manual.pdf'
warning: no files found matching '*' under directory 'doc/html'
warning: no files found matching 'somoclu/m4/*'
writing manifest file 'somoclu.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_py
running build_ext
building '_somoclu_wrap' extension
gcc -fno-strict-aliasing -I/Users/fresco/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/src/denseCpuKernels.cpp -o build/temp.macosx-10.7-x86_64-2.7/somoclu/src/denseCpuKernels.o -fopenmp
gcc -fno-strict-aliasing -I/Users/fresco/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/src/sparseCpuKernels.cpp -o build/temp.macosx-10.7-x86_64-2.7/somoclu/src/sparseCpuKernels.o -fopenmp
gcc -fno-strict-aliasing -I/Users/fresco/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/src/mapDistanceFunctions.cpp -o build/temp.macosx-10.7-x86_64-2.7/somoclu/src/mapDistanceFunctions.o -fopenmp
gcc -fno-strict-aliasing -I/Users/fresco/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/src/training.cpp -o build/temp.macosx-10.7-x86_64-2.7/somoclu/src/training.o -fopenmp
gcc -fno-strict-aliasing -I/Users/fresco/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/src/uMatrix.cpp -o build/temp.macosx-10.7-x86_64-2.7/somoclu/src/uMatrix.o -fopenmp
gcc -fno-strict-aliasing -I/Users/fresco/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/somoclu_wrap.cxx -o build/temp.macosx-10.7-x86_64-2.7/somoclu/somoclu_wrap.o -fopenmp
/Developer/NVIDIA/CUDA-7.5/bin/nvcc -DCUDA -I/Users/fresco/anaconda/lib/python2.7/site-packages/numpy/core/include -Isrc -I/Developer/NVIDIA/CUDA-7.5/include -I/Users/fresco/anaconda/include/python2.7 -c somoclu/src/denseGpuKernels.cu -o build/temp.macosx-10.7-x86_64-2.7/somoclu/src/denseGpuKernels.o -use_fast_math --ptxas-options=-v -c --compiler-options -fPIC -fopenmp
nvcc fatal   : The version ('60000') of the host compiler ('clang') is not supported
Traceback (most recent call last):
  File "setup.py", line 173, in <module>
    test_suite="tests"
  File "/Users/fresco/anaconda/lib/python2.7/distutils/core.py", line 166, in setup
    raise SystemExit, "error: " + str(msg)
SystemExit: error: command '/Developer/NVIDIA/CUDA-7.5/bin/nvcc' failed with exit status 1
running install
running bdist_egg
running egg_info
writing requirements to somoclu.egg-info/requires.txt
writing somoclu.egg-info/PKG-INFO
writing top-level names to somoclu.egg-info/top_level.txt
writing dependency_links to somoclu.egg-info/dependency_links.txt
reading manifest file 'somoclu.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'doc/manual.pdf'
warning: no files found matching '*' under directory 'doc/html'
warning: no files found matching 'somoclu/m4/*'
writing manifest file 'somoclu.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_py
creating build/bdist.macosx-10.7-x86_64
creating build/bdist.macosx-10.7-x86_64/egg
creating build/bdist.macosx-10.7-x86_64/egg/somoclu
copying build/lib/somoclu/somoclu_wrap.py -> build/bdist.macosx-10.7-x86_64/egg/somoclu
copying build/lib/somoclu/train.py -> build/bdist.macosx-10.7-x86_64/egg/somoclu
copying build/lib/somoclu/__init__.py -> build/bdist.macosx-10.7-x86_64/egg/somoclu
byte-compiling build/bdist.macosx-10.7-x86_64/egg/somoclu/somoclu_wrap.py to somoclu_wrap.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/somoclu/train.py to train.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/somoclu/__init__.py to __init__.pyc
creating build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying somoclu.egg-info/PKG-INFO -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying somoclu.egg-info/SOURCES.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying somoclu.egg-info/dependency_links.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying somoclu.egg-info/requires.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying somoclu.egg-info/top_level.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
somoclu.somoclu_wrap: module references __file__
creating dist
creating 'dist/somoclu-1.7.5-py2.7.egg' and adding 'build/bdist.macosx-10.7-x86_64/egg' to it
removing 'build/bdist.macosx-10.7-x86_64/egg' (and everything under it)
Processing somoclu-1.7.5-py2.7.egg
creating /Users/fresco/anaconda/lib/python2.7/site-packages/somoclu-1.7.5-py2.7.egg
Extracting somoclu-1.7.5-py2.7.egg to /Users/fresco/anaconda/lib/python2.7/site-packages
Adding somoclu 1.7.5 to easy-install.pth file

Installed /Users/fresco/anaconda/lib/python2.7/site-packages/somoclu-1.7.5-py2.7.egg
Processing dependencies for somoclu==1.7.5
Searching for scipy==0.19.0
Best match: scipy 0.19.0
Adding scipy 0.19.0 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for matplotlib==2.0.2
Best match: matplotlib 2.0.2
Adding matplotlib 2.0.2 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for numpy==1.12.1
Best match: numpy 1.12.1
Adding numpy 1.12.1 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for pyparsing==2.1.4
Best match: pyparsing 2.1.4
Adding pyparsing 2.1.4 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for cycler==0.10.0
Best match: cycler 0.10.0
Adding cycler 0.10.0 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for pytz==2017.2
Best match: pytz 2017.2
Adding pytz 2017.2 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for subprocess32==3.2.7
Best match: subprocess32 3.2.7
Adding subprocess32 3.2.7 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for functools32==3.2.3.post2
Best match: functools32 3.2.3.post2
Adding functools32 3.2.3.post2 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for python-dateutil==2.6.0
Best match: python-dateutil 2.6.0
Adding python-dateutil 2.6.0 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Searching for six==1.10.0
Best match: six 1.10.0
Adding six 1.10.0 to easy-install.pth file

Using /Users/fresco/anaconda/lib/python2.7/site-packages
Finished processing dependencies for somoclu==1.7.5```
peterwittek commented 6 years ago

We discussed macOS-related problems and the current solution in gory details in this issue. The only thing we can do is to ensure that it passes the OSX build with Travis CI with conda (see the corresponding recipe.

xgdgsc commented 6 years ago

Maybe you should try install the same version of numpy that conda forge uses.

jazoza commented 6 years ago

Indeed, @xgdgsc what helped was to do a clean reinstall of conda on the desktop machine and get all packages versions in sync I guess. Now I am able to run the example code. Thank you both!