search

cosmicrays / hermes

HERMES is a publicly available computational framework for the line of sight integration over galactic radiative processes which creates sky maps in the HEALPix-compatibile format.
GNU General Public License v3.0
22 stars 9 forks source link

pyhermes module not found #22

Open tospines opened 2 years ago

tospines commented 2 years ago

Hello! Python is having troubles to find the pyhermes module, even though I am importing pyhermes in the build folder, where the pyhermes.cpython-38-x86_64-linux-gnu.so is located. I have tried adding explicitly the build path to the PYTHONPATH variable and to the python sys.path(), but for both cases I get the same error:

import pyhermes Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'pyhermes'

I am installing Hermes in a cluster of computers, within an environment where I have full control, and there was no problem in the installation process.

Any ideas on how to solve this? Thanks!

adundovi commented 2 years ago

Hi @tospines, do you use virtualenv to install Hermes on Linux as described here?

tospines commented 2 years ago

Yes, I am totally following the same steps as described there. Curiously I do not get any error or strange warning in the installation step, but then, it does not find the pyhermes package.

adundovi commented 2 years ago

Here are the steps which should be checked for the Python binding to work:

  1. During the cmake phase one should ensure that the cmake outputs the following (exact versions are not important):

    • Found PythonInterp: ~/.virtualenvs/hermes/bin/python (found version "3.10.2")
    • Found PythonLibs: /usr/lib64/libpython3.10.so (found version "3.10.2")
    • Python: Found! -- Version: 3.10/310 -- Executeable: ~/.virtualenvs/hermes/bin/python -- Include: /usr/include/python3.10 -- Library: /usr/lib64/libpython3.10.so -- Site-package directory: ~/.virtualenvs/hermes/lib64/python3.10/site-packages

  2. The build phase should finish successfully, of course, and it has to contain the following line:

    • [100%] Linking CXX shared module pyhermes.cpython-310-x86_64-linux-gnu.so

  3. Testing: with make test one can check all C++ parts of the code where all tests should pass, however, to test the Python binding one can run python interpreter within the build directory where pyhermes.cpython-310-x86_64-linux-gnu.so is present - and there one should be able to import pyhermes: 

    $ python
Python 3.10.2 (main, Jan 17 2022, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyhermes
>>> dir(pyhermes)
['BremsstrahlungIntegrator', 'CircularWindow', 'DarkMatterIntegrator',  (...), 'units']

  4. If the previous step works, one can finally execute make install which will copy all compiled binaries and resource files to ~/.virtualenvs/hermes/ and one can use pyhermes module from any path as long as the given virtualenv is active.

This has been trying many times on various machines, so please double-check all of these and report back if you have some strange outputs or errors.

tospines commented 2 years ago

I have followed those steps and found a couple of differences with respect what you described:

Then, after the make -j step, I got the "[100%] Linking CXX shared module pyhermes.cpython-310-x86_64-linux-gnu.so" message as you said. Unfortunately, once I run the make test I get an error at the end: --The following tests FAILED: 26 - testPixelTimes (Failed)

I am attaching to this message the LastTest.log in case it can help. LastTest.log LastTestsFailed.log

Finally, I tried to import pyhermes but it did not work, as earlier.

Should I link the Python interpreter in the cmake step or something like this?

adundovi commented 2 years ago

26 - testPixelTimes (Failed)

OK, this test can fail as it only measures performance, i.e., pixel calculation times for various integrators to check for performance regressions when introducing new features or changing the code. These times are somehow manually picked by me to match some reasonably recent desktop hardware, but on laptops, these times could be inadequate.

Regarding your issue of loading pyhermes in Python - it looks to me that you have several Python versions installed on your system and with one version your pyhermes module is compiled, while you run the interpreter where you try to load it with the other one.

To understand which Python interpreter you run, please give me the output of: $ python -V. Have you activated a virtual environment before running cmake? You could manually specify the Python paths (PYTHON_INCLUDE_PATH and PYTHON_LIBRARIES) with -D flag in cmake.

tospines commented 2 years ago

Yes, you are right, there are many Python versions installed here. The one I have used for installing and, then, for trying to import pyhermes is Python 3.6.7 (this is the output of python -V). As I am working in a cluster of computers I first activate an environment and, then, the Hermes environment. Once I am within the HERMES environment, I run the cmake.

adundovi commented 2 years ago

OK, but you mentioned in your first message a file named pyhermes.cpython-38-x86_64-linux-gnu.so from what I read that Python 3.8 (38) is found during the cmake phase and this will certainly not work with Python 3.6.7. So, either use Python 3.8 within your virtual environment or instruct cmake to find 3.6.7.

tospines commented 2 years ago

Yo were right, it seemed that the python version that I had changed when I entered in my environment. Solved now! Thank you very much.

fayez0 commented 1 year ago

Hello!

I am facing the same problem here. I've installed Hermes on Mac OS. I used a virtualenv to install and run it, but I still have the same problem. When I run cmak test, I get this error:

The following tests FAILED: 9 - testIntegrationMethods (Subprocess aborted) Errors while running CTest Output from these tests are in: /Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/Testing/Temporary/LastTest.log Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. make: *** [test] Error 8 (base) fayezabu-ajamieh@Fayezs-MacBook-Pro build % --rerun-failed --output-on-failure zsh: command not found: --rerun-failed

adundovi commented 1 year ago

Hi @fayez0, please add the last log file (/Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/Testing/Temporary/LastTest.log) to provide more details. //Andrej

fayez0 commented 1 year ago

Here it is LastTest.log

fayez0 commented 1 year ago

Any idea why I keep getting this error? It seems the code cannot find pyhermes

fayez0 commented 1 year ago

Here is the portion of the test that has the error:

9/28 Testing: testIntegrationMethods 9/28 Test: testIntegrationMethods Command: "/Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build/testIntegrationMethods" Directory: /Users/fayezabu-ajamieh/Desktop/Study/Research/PhD_Research/tools/hermes/build "testIntegrationMethods" start time: Apr 03 10:32 IST Output:

dyld[20643]: symbol not found in flat namespace '_cblas_caxpy'

Test time = 0.01 sec ---------------------------------------------------------- Test Failed. "testIntegrationMethods" end time: Apr 03 10:32 IST "testIntegrationMethods" time elapsed: 00:00:00 ----------------------------------------------------------
carmeloevoli commented 1 year ago

Could you try to compile it and test it without python first? Just to be sure that the C++ part works properly. Use -DENABLE_PYTHON=OFF

fayez0 commented 1 year ago

I got this:

/Library/Developer/CommandLineTools/usr/bin/make: invalid option -- D /Library/Developer/CommandLineTools/usr/bin/make: invalid option -- E Usage: make [options] [target] ... Options: -b, -m Ignored for compatibility. -B, --always-make Unconditionally make all targets. -C DIRECTORY, --directory=DIRECTORY Change to DIRECTORY before doing anything. -d Print lots of debugging information. --debug[=FLAGS] Print various types of debugging information. -e, --environment-overrides Environment variables override makefiles. -f FILE, --file=FILE, --makefile=FILE Read FILE as a makefile. -h, --help Print this message and exit. -i, --ignore-errors Ignore errors from commands. -I DIRECTORY, --include-dir=DIRECTORY Search DIRECTORY for included makefiles. -j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg. -k, --keep-going Keep going when some targets can't be made. -l [N], --load-average[=N], --max-load[=N] Don't start multiple jobs unless load is below N. -L, --check-symlink-times Use the latest mtime between symlinks and target. -n, --just-print, --dry-run, --recon Don't actually run any commands; just print them. -o FILE, --old-file=FILE, --assume-old=FILE Consider FILE to be very old and don't remake it. -p, --print-data-base Print make's internal database. -q, --question Run no commands; exit status says if up to date. -r, --no-builtin-rules Disable the built-in implicit rules. -R, --no-builtin-variables Disable the built-in variable settings. -s, --silent, --quiet Don't echo commands. -S, --no-keep-going, --stop Turns off -k. -t, --touch Touch targets instead of remaking them. -v, --version Print the version number of make and exit. -w, --print-directory Print the current directory. --no-print-directory Turn off -w, even if it was turned on implicitly. -W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE Consider FILE to be infinitely new. --warn-undefined-variables Warn when an undefined variable is referenced. -N OPTION, --NeXT-option=OPTION Turn on value of NeXT OPTION.

This program built for i386-apple-darwin11.3.0 Report bugs to bug-make@gnu.org

By the way, I tried to installed on a remote machine via ssh. It passed all tests, but I still the error:

from pyhermes import * Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'pyhermes'

carmeloevoli commented 1 year ago

please check it carefully, it cannot be invalid argument:

mkdir build
cd build
cmake  -DENABLE_COVERAGE=OFF -DENABLE_PYTHON=OFF -DENABLE_TESTING=ON ..
fayez0 commented 1 year ago

Same issue

adundovi commented 1 year ago

dyld[20643]: symbol not found in flat namespace '_cblas_caxpy'

This is an issue, and it seems it is related to GSL and new OS X - look here for a possible solution.

adundovi commented 1 year ago

Maybe reinstalling GLS or BLAS (openblas from homebrew?) and repeating the hermes compilation could fix the issue.

BTW, your issue is unrelated to the first one, for which this thread was created, as in your case the pyhermes module is successfully compiled and all of tests run OK except that one. Hence, in this case, it would be better to open a new ticket.

fayez0 commented 1 year ago

This doesn't seem to be the problem: I tried to install hermes on a cluster with Ubuntu 22 using ssh. It installs perfectly and when I run make test, I get:

.... 100% tests passed, 0 tests failed out of 28

Total Test time (real) = 12.12 sec

However, I still have the same error.

(hermes) fayez@sudhir-linux-server:~/hermes/build$ python Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

from pyhermes import * Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'pyhermes'

gr-soawnit commented 10 months ago

I have problem , when I import pyhermes ImportError: /home/hence/hermes/build/pyhermes.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN6hermes11nside2orderEj

make -j [100%] Built target pyhermes -- Build Type: Release -- Use --as-needed linker flags! -- Use absolute RPATH /home/hence/hermes/lib -- Downloading data file from heat.gssi.it ~ 200 MB -- Extracting data file -- Found PythonInterp: /home/hence/anaconda3/envs/hermes/bin/python (found version "3.11.5") -- FFTW3F header => /home/hence/anaconda3/envs/hermes/include -- FFTW3F libs => /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so -- FFTW3 with single precision (FFTW3F): Found! -- Include: /home/hence/anaconda3/envs/hermes/include -- Library: /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so -- found version string 4.3 -- Found CFITSIO 4.3: /home/hence/anaconda3/envs/hermes/lib/libcfitsio.so -- HDF5: Using hdf5 compiler wrapper to determine C configuration -- Parallel hdf5: FALSE -- Hermes version: v1.0.1-96-g165cd2ac 165cd2ac25baf684590750f7e9ca7f551e613eb3 refs/heads/master Enabling coverage report via $make coverage Units dimensionality check enabled. -- Python: Found! -- Version: 3.11/311 -- Executeable: /home/hence/anaconda3/envs/hermes/bin/python -- Include: /home/hence/anaconda3/envs/hermes/include/python3.11 -- Library: /home/hence/anaconda3/envs/hermes/lib/libpython3.11.so -- Site-package directory: /home/hence/anaconda3/envs/hermes/lib/python3.11/site-packages -- pybind11 v2.11.0 dev1 -- Found Doxygen and Sphinx to build documentation -- Configuring done -- Generating done -- Build files have been written to: /home/hence/hermes/build (hermes) hence@DESKTOP-M6C1FJR:~/hermes/build$ CMAKE_PREFIX_PATH=$HERMES_DIR cmake -DCMAKE_INSTALL_PREFIX=$HERMES_DIR -DENABLE_TESTING=On .. -- Build Type: Release -- Use --as-needed linker flags! -- Use absolute RPATH /home/hence/hermes/lib -- Downloading data file from heat.gssi.it ~ 200 MB -- Extracting data file -- FFTW3F header => /home/hence/anaconda3/envs/hermes/include -- FFTW3F libs => /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so -- FFTW3 with single precision (FFTW3F): Found! -- Include: /home/hence/anaconda3/envs/hermes/include -- Library: /home/hence/anaconda3/envs/hermes/lib/libfftw3f.so -- found version string 4.3 -- Found CFITSIO 4.3: /home/hence/anaconda3/envs/hermes/lib/libcfitsio.so -- HDF5: Using hdf5 compiler wrapper to determine C configuration -- Parallel hdf5: FALSE -- Hermes version: v1.0.1-96-g165cd2ac 165cd2ac25baf684590750f7e9ca7f551e613eb3 refs/heads/master Enabling coverage report via $make coverage Units dimensionality check enabled. -- Python: Found! -- Version: 3.11/311 -- Executeable: /home/hence/anaconda3/envs/hermes/bin/python -- Include: /home/hence/anaconda3/envs/hermes/include/python3.11 -- Library: /home/hence/anaconda3/envs/hermes/lib/libpython3.11.so -- Site-package directory: /home/hence/anaconda3/envs/hermes/lib/python3.11/site-packages -- pybind11 v2.11.0 dev1 -- Found Doxygen and Sphinx to build documentation -- Configuring done -- Generating done -- Build files have been written to: /home/hence/hermes/build

adundovi commented 10 months ago

Hi @gr-soawnit, which version/distribution of Linux do you use?

gr-soawnit commented 9 months ago

I so sorry for late response, I try in Ubuntu 20.04 and 22.04

On Sun, Nov 26, 2023 at 12:07 AM Andrej Dundovic @.***> wrote:

Hi @gr-soawnit https://github.com/gr-soawnit, which version/distribution of Linux do you use?

— Reply to this email directly, view it on GitHub https://github.com/cosmicrays/hermes/issues/22#issuecomment-1826378738, or unsubscribe https://github.com/notifications/unsubscribe-auth/BEGYPAMQSMT4UYOFTEV4MB3YGIQSPAVCNFSM5PVSOHL2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBSGYZTOOBXGM4A . You are receiving this because you were mentioned.Message ID: @.***>

gr-soawnit commented 8 months ago

I try to fix the problem. Now I got [ 5%] Built target cparamlib [ 8%] Built target kiss [ 46%] Built target hermes [ 59%] Built target pyhermes [ 60%] Built target gtest [ 61%] Built target gtest_main [ 62%] Built target testPicard3D [ 63%] Built target testHdf5Reader [ 65%] Built target testUnits [ 66%] Built target testCommon [ 68%] Built target testNumericalIntegration [ 70%] Built target testCacheTools [ 71%] Built target testVector3 [ 73%] Built target testHEALPix [ 74%] Built target testIntegrationMethods [ 76%] Built target testSkymap [ 77%] Built target testFITS [ 78%] Built target testCosmicRays [ 79%] Built target testYMW16 [ 81%] Built target testDispersionMeasureIntegrator [ 82%] Built target testRotationMeasureIntegrator [ 83%] Built target testSynchroIntegrator [ 84%] Built target testFreeFreeIntegrator [ 85%] Built target testMFieldModels [ 86%] Built target testSynchroAbsorptionIntegrator [ 88%] Built target testInteractions [ 89%] Built target testRingModel [ 90%] Built target testPiZeroIntegrator [ 91%] Built target testBremsstrahlungIntegrator [ 92%] Built target testPhotonField [ 93%] Built target testInverseComptonIntegrator [ 94%] Built target testBreitWheeler [ 95%] Built target testDarkMatter [ 97%] Built target testPixelTimes [100%] Built target test_uuid Install the project... -- Install configuration: "Release" -- Installing: /home/codex/anaconda3/envs/hermes/lib/python3.12/site-packages/pyhermes.cpython-312-x86_64-linux-gnu.so -- Set runtime path of "/home/codex/anaconda3/envs/hermes/lib/python3.12/site-packages/pyhermes.cpython-312-x86_64-linux-gnu.so" to "" -- Installing: /lib/libhermes.so CMake Error at cmake_install.cmake:80 (file): file INSTALL cannot copy file "/home/codex/hermes/build/libhermes.so" to "/lib/libhermes.so": Permission denied.

I cann't do with sudo make install sudo: make: command not found

adundovi commented 8 months ago

Hi, @gr-soawnit, this error means that you didn't specified a custom install path and the default system one (/lib/) is used for which you need the root permission to copy into.

-- Installing: /lib/libhermes.so
CMake Error at cmake_install.cmake:80 (file):
file INSTALL cannot copy file "/home/codex/hermes/build/libhermes.so" to
"/lib/libhermes.so": Permission denied.

You can read easily this output: "...trying to install /lib/libhermes.so by copying the file from /home/codex/hermes/build/libhermes.so... err, unfortunately, didn't work: permission denied." So, either you can run cmake install with sudo in front to run it with the admin permission, which will certainly copy it to /lib. Or you can repeat the step cmake .. where you should use -DCMAKE_INSTALL_PREFIX=$HERMES_DIR. And $HERMES_DIR is an environment variable where you would like to install Hermes in some directory where you have permission to copy files into such as, HERMES_DIR=$HOME"/.virtualenvs/hermes". Please, see "Install on GNU/Linux" in INSTALL.

gr-soawnit commented 8 months ago

Hi, @adundovi , I try to solve this follow your advice but it already still have same problem ImportError: /home/hence/hermes/build/pyhermes.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN6hermes11nside2orderEj

now, I install in ubuntu 22.24, another partition, it got back to problem "ModuleNotFoundError: No module named 'pyhermes'" I follow to your installation, I got

CMAKE_PREFIX_PATH=$HERMES_DIR cmake -DCMAKE_INSTALL_PREFIX=$HERMES_DIR -DENABLE_COVERAGE=ON -DENABLE_PYTHON=ON -DENABLE_TESTING=ON ..

-- Build Type: Release -- Use --as-needed linker flags! -- Use absolute RPATH /home/codex/.virtualenvs/hermes/lib -- Downloading data file from heat.gssi.it ~ 200 MB -- Extracting data file -- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter -- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed -- FFTW3F header => /usr/include -- FFTW3F libs => /usr/lib/x86_64-linux-gnu/libfftw3f.so -- FFTW3 with single precision (FFTW3F): Found! -- Include: /usr/include -- Library: /usr/lib/x86_64-linux-gnu/libfftw3f.so -- found version string 4.0 -- Found CFITSIO 4.0: /usr/lib/x86_64-linux-gnu/libcfitsio.so -- Could NOT find HDF5 (missing: HDF5_LIBRARIES HDF5_INCLUDE_DIRS) (found version "") -- Hermes version: v1.0.1-97-g16299515 16299515303a9e709c8a31c5310a2dbd7a303bb2 refs/heads/master Enabling coverage report via $make coverage Units dimensionality check enabled. -- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter -- Found Python3: /home/codex/.virtualenvs/hermes/bin/python3.10 (found version "3.10.12") found components: Interpreter Development Development.Module Development.Embed -- Found Doxygen and Sphinx to build documentation -- Configuring done -- Generating done -- Build files have been written to: /home/codex/hermes/build

gr-soawnit commented 8 months ago

I install python 3.10.12 for new installation

adundovi commented 8 months ago

ModuleNotFoundError: No module named 'pyhermes'" could mean two things: either no python-dev package was found during the cmake procedure, or there is a mistake in Python paths. In both cases Python will not be able to find pyhermes.

Do you know how to use Docker? I can prepare a docker image with jupyter and HERMES which should straightforward to install?

gr-soawnit commented 8 months ago

@adundovi I already have python-dev but don't sure, this cann't find path

Docker image, It okay. I have used it before, but I'm not familiar with it. it will be helpful.

adundovi commented 8 months ago

@gr-soawnit, I updated the code with two things:

  1. I have cleaned up and updated the CMake code for finding Python, so I hope it will work better with newer versions of various software stacks;
  2. Now, there is a Dockerfile with which you can easily build and run a docker image with Jupyter and HERMES installed: 
    docker build -t jupyter-pyhermes .
docker run --name jupyter -it --rm -p 8888:8888 -v $HOME/jupyter-work:/home/jovyan/work localhost/jupyter-pyhermes:latest

    See the bottom of INSTALL for more information.