search

gimli-org / gimli

Geophysical Inversion and Modeling Library :earth_africa:
https://www.pygimli.org
Other
372 stars 136 forks source link

Failure to install on mac #730

Open srkaeppler opened 4 months ago

srkaeppler commented 4 months ago

The documentation has gotten a lot better on installation. I decided to attempt to install on mac using brew and venv. I realize this is not supported, but I have wanted to stay away from conda because I heard that it is hard to remove from your computer, although I may go that route. I was able to get completions of: make -j=8 and make pygimli J=8 However, when I test the code by loading the code, I get the following error:

(gimli) srkaeppler@Stephens-MBP-2 build % make check
[ 95%] Built target gimli
[ 95%] Building CXX object core/tests/unittest/CMakeFiles/gimliUnitTest.dir/unitTest.cpp.o
In file included from /Users/srkaeppler/research/data/pygimli/source/core/tests/unittest/unitTest.cpp:11:
/Users/srkaeppler/research/data/pygimli/source/core/tests/unittest/testVector.h:468:17: warning: unused variable '_AT' [-Wunused-variable]
        double *_AT = new double[k*m];
                ^
1 warning generated.
[100%] Linking CXX executable ../../../bin/gimliUnitTest
[100%] Built target gimliUnitTest
Test project /Users/srkaeppler/research/data/pygimli/source/build
    Start 1: gtest:gimli_unit_tests
1/2 Test #1: gtest:gimli_unit_tests ...........***Failed    0.47 sec
    Start 2: pgtest:pygimli_tests
2/2 Test #2: pgtest:pygimli_tests .............***Failed    0.06 sec

0% tests passed, 2 tests failed out of 2

Total Test time (real) =   0.53 sec

The following tests FAILED:
      1 - gtest:gimli_unit_tests (Failed)
      2 - pgtest:pygimli_tests (Failed)
Errors while running CTest
Output from these tests are in: /Users/srkaeppler/research/data/pygimli/source/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make[3]: *** [CMakeFiles/check] Error 8
make[2]: *** [CMakeFiles/check.dir/all] Error 2
make[1]: *** [CMakeFiles/check.dir/rule] Error 2
make: *** [check] Error 2

Start testing: May 29 14:57 EDT
----------------------------------------------------------
1/2 Testing: gtest:gimli_unit_tests
1/2 Test: gtest:gimli_unit_tests
Command: "/usr/bin/make" "gtest"
Directory: /Users/srkaeppler/research/data/pygimli/source/build/core/tests/unittest
"gtest:gimli_unit_tests" start time: May 29 14:57 EDT
Output:
----------------------------------------------------------
[ 95%] Built target gimli
[100%] Built target gimliUnitTest
dyld[41987]: missing symbol called
/bin/sh: line 1: 41987 Abort trap: 6           ../../../bin/gimliUnitTest
make[7]: *** [gtest] Error 134
make[6]: *** [core/tests/unittest/CMakeFiles/gtest.dir/all] Error 2
make[5]: *** [core/tests/unittest/CMakeFiles/gtest.dir/rule] Error 2
make[4]: *** [core/tests/unittest/CMakeFiles/gtest.dir/rule] Error 2
<end of output>
Test time =   0.47 sec
----------------------------------------------------------
Test Failed.
"gtest:gimli_unit_tests" end time: May 29 14:57 EDT
"gtest:gimli_unit_tests" time elapsed: 00:00:00
----------------------------------------------------------

2/2 Testing: pgtest:pygimli_tests
2/2 Test: pgtest:pygimli_tests
Command: "/usr/bin/make" "pgtest"
Directory: /Users/srkaeppler/research/data/pygimli/source/build/core/python
"pgtest:pygimli_tests" start time: May 29 14:57 EDT
Output:
----------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pygimli'
make[7]: *** [pgtest] Error 1
make[6]: *** [core/python/CMakeFiles/pgtest.dir/all] Error 2
make[5]: *** [core/python/CMakeFiles/pgtest.dir/rule] Error 2
make[4]: *** [core/python/CMakeFiles/pgtest.dir/rule] Error 2
<end of output>
Test time =   0.06 sec
----------------------------------------------------------
Test Failed.
"pgtest:pygimli_tests" end time: May 29 14:57 EDT
"pgtest:pygimli_tests" time elapsed: 00:00:00
----------------------------------------------------------

End testing: May 29 14:57 EDT

(gimli) srkaeppler@Stephens-MBP-2 source % python3 -c 'import pygimli'
Traceback (most recent call last):
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/core/core.py", line 11, in <module>
    from . import _pygimli_  # if it works: as pgcore, replace all _pygimli_
    ^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/Users/srkaeppler/research/data/pygimli/source/pygimli/core/_pygimli_.so, 0x0002): symbol not found in flat namespace '_openblas_set_num_threads'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/__init__.py", line 8, in <module>
    from .core.decorators import (renamed, singleton, moduleProperty,
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/core/__init__.py", line 11, in <module>
    from .core import pgcore
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/core/core.py", line 19, in <module>
    import pgcore
ModuleNotFoundError: No module named 'pgcore'

I am not sure what the issue is.

My brew and enviornment installation includes the following packages:

(gimli) srkaeppler@Stephens-MacBook-Pro-2 build % brew list
==> Formulae
boost           gsl         mercurial       python@3.11
boost-python3       hwloc           metis           python@3.12
ca-certificates     icu4c           mpdecimal       readline
cmake           isl         mpfr            sqlite
fftw            libevent        open-mpi        suite-sparse
gcc         libidn2         openblas        triangle
gdbm            libmpc          openssl@3       wget
gettext         libomp          pcre2           xz
git         libunistring        pmix            zstd
gmp         lz4         python-gdbm@3.12

(gimli) srkaeppler@Stephens-MacBook-Pro-2 build % pip freeze
contourpy==1.2.1
cycler==0.12.1
fonttools==4.52.4
kiwisolver==1.4.5
matplotlib==3.9.0
numpy==1.26.4
packaging==24.0
pillow==10.3.0
pygccxml==2.5.0
pyparsing==3.1.2
pyplusplus==1.8.5
python-dateutil==2.9.0.post0
scipy==1.13.1
setuptools==70.0.0
six==1.16.0

Here is the full output in this file: https://www.dropbox.com/scl/fi/7antaqtoafg3ra3tsmgms/gimli-srkaeppler-Stephens-MBP-2-pygiml.txt?rlkey=rdlr69962pl34qwibilcjay22&dl=0

florian-wagner commented 4 months ago

Dear @srkaeppler,

not sure why the test target fails, but the installation generally seems to have worked.

Can you check if the following works

cd /Users/srkaeppler/research/data/pygimli/source/
python -c "import pygimli; print(pygimli.__version__)"

If yes, you probably just need to add the path to your .bashrc.

Best wishes Florian

srkaeppler commented 4 months ago

@florian-wagner : 

srkaeppler@Stephens-MacBook-Pro-2 foo % source ~srkaeppler/Env3/gimli/bin/activate
(gimli) srkaeppler@Stephens-MacBook-Pro-2 foo % cd ~srkaeppler/research/data/pygimli/source 
(gimli) srkaeppler@Stephens-MacBook-Pro-2 source % ls
AUTHORS.rst              INSTALLATION.rst         apps                     environment.yml
CHANGELOG.md             ISSUE_TEMPLATE.md        build                    pygimli
CITATION.cff             MANIFEST.in              core                     pygimli.egg-info
CITATION.rst             PULL_REQUEST_TEMPLATE.md dependencies.rst         setup.cfg
CMakeLists.txt           README.md                dev_requirements.txt     setup.py
CONTRIBUTING.md          README.rst               dist                     versioneer.py
COPYING.rst              __pycache__              doc
(gimli) srkaeppler@Stephens-MacBook-Pro-2 source % python -c "import pygimli; print(pygimli.__version__)"
Traceback (most recent call last):
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/core/core.py", line 11, in <module>
    from . import _pygimli_  # if it works: as pgcore, replace all _pygimli_
    ^^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/Users/srkaeppler/research/data/pygimli/source/pygimli/core/_pygimli_.so, 0x0002): symbol not found in flat namespace '_openblas_set_num_threads'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/__init__.py", line 8, in <module>
    from .core.decorators import (renamed, singleton, moduleProperty,
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/core/__init__.py", line 11, in <module>
    from .core import pgcore
  File "/Users/srkaeppler/research/data/pygimli/source/pygimli/core/core.py", line 19, in <module>
    import pgcore
ModuleNotFoundError: No module named 'pgcore'
(gimli) srkaeppler@Stephens-MacBook-Pro-2 source %

I also very much appreciate the help. I have debated whether to just use conda and bite the bullet. Brady got me onto this code and I have a particular nonlinear regularization problem I am wondering if this code could help with, so I am especially interested in the regularization notebooks, etc.

carsten-forty2 commented 4 months ago

After some recent changes on the build toolchain its good to remove the CMakeCache.txt in the build path or even try a new clean build.

You could also give the wheel branch a try. Its on the same state like the dev branch plus a test for a build-chain for whl generation using venv. Note, this branch is not yet tested for mac and there might be some minor tweeks necessary.

In the new branch are new make targets which should produce valid whl files for local pip installation.

make whlpgcore

make whlpygimli

Both create there respective whl files and test them against a local test venv. The final whl's you can find in build/wheelhouse. If there is something wrong maybe the cmake output give some hints.

srkaeppler commented 4 months ago

@carsten-forty2 @florian-wagner Thanks for the response. I see two potential actions here and let me outline the proposed procedure.

One:

  1. Reclone pygimli?
  2. rebuild following the procedure outlined above. I am not really sure this will work unless some sort of flag has been changed in the Cmake files. This is something I am not familiar with writing. Put another way, I could basically do this right now and I have tried this by deleting the build folder and unless some other files have changed (I could git pull), I don't see why I would get a different outcome

Two:

  1. Reclone gimli
  2. git checkout wheel to get onto the wheel branch.
  3. make -j=8
  4. make whlpgcore
  5. make whlpygimli
  6. make pygimli J=8 Is this the procedure?

One last thing, how do I build or make pgcore maybe that is what is screwing up...

srkaeppler commented 4 months ago

@carsten-forty2 @florian-wagner The other issue I am still concerned about is that the .so file above has some missing tags. I am not sure if this is something you need to set as a compiler option or some sort of .bashrc environment variable. I am setting some of this stuff in .zshrc since I am using a mac with a M3 chip. I have not extensively worked with openblas or any of those packages, so any help there would be appreciated.

Thank you both for helping me! Once we get this working, I am happy to post this to you in some form and you could potentially add it to your website.

srkaeppler commented 4 months ago

Just recloned this morning and getting new errors during the make install -j8 step:

  file INSTALL cannot find
  "/Users/srkaeppler/research/data/pygimli/source/build/gimli_export.h": No
  such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:42 (include)