search

INCF / MUSIC

MUSIC, the MUltiSimulation Coordinator
GNU General Public License v3.0
37 stars 37 forks source link

Pymusic issues #10

Open uahic opened 8 years ago

uahic commented 8 years ago

I've compiled the recent music version and tried to import it with Ipython, the first error Ive got is that when it enteres the init.py of the music package in the build path, it cannot find pymusic as it is imported 'absolute'. So I just modified the line to from .pymusic import *

After that another issue arises where I am not sure if I am using the package in a wrong manner. Ive tried it also to use with mpiexec ipython on a cluster (openmpi 1.8.7, mpi4py 2.0.0). The python interpreter is from a virtual environment for python3.4 (freshly setup with Cython and mpi4py), MUSIC ENV's are set as follows:

export MUSIC_ROOT_DIR=/gpfs/bbp.cscs.ch/home/bbpnrsoa/martin.setup/builds/MUSIC_1_1_15 export MUSIC_ROOT=${MUSIC_ROOT_DIR} MUSIC_PATH=${MUSIC_ROOT_DIR} export LD_LIBRARY_PATH=${MUSIC_PATH}/lib:$LD_LIBRARY_PATH export PATH=${MUSIC_PATH}/bin:$PATH export CPATH=${MUSIC_PATH}/include:$CPATH export PYTHONPATH=${MUSIC_PATH}/lib/python3.4/site-packages:$PYTHONPATH

` iPython 3.4.3 (default, Jan 18 2016, 16:48:08) Type "copyright", "credits" or "license" for more information.

IPython 4.1.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.

import music

ValueError Traceback (most recent call last)

in () ----> 1 import music /gpfs/bbp.cscs.ch/home/bbpnrsoa/martin.setup/builds/MUSIC_1_1_15/lib/python3.4/site-packages/music/**init**.py in () 23 else: 24 setrc() ---> 25 from .pymusic import * 26 27 #import DLFCN as dl MPI.pxd in init pymusic (pymusic.cpp:12172)() ValueError: mpi4py.MPI.Win has the wrong size, try recompiling In [2]: ` As far as I can recall from the documentation MPI.Win was the windows size for buffers (mpi3 features?) but no idea why
apeyser commented 8 years ago

I expect that mpi4py was compiled with a different mpi than than music -- does your environment have a module system? You may need to do a module load xxxx before building and/or running music.

For the .pymusic --- yes, pymusic needs to be updated to python3+

uahic commented 8 years ago

I was not sure if I really compiled everything with the same mpi module (yep we do have the module system on the cluster) but now I recompiled everything and made sure that the make-scripts of music and mpi4py take the same openmpi-1.8.7 package. The issue still exists, so probably I need to have a look in the music implementation and if the problem is not there to ask the mpi4py guy. Mpi4py seems to work so far without any complaints

uahic commented 8 years ago

I found that people had the same error message (with swig or cython wrappers) in the past, so I posted that issue to the mpi4py issue-tracker, here is a link to follow up the current status:

https://bitbucket.org/mpi4py/mpi4py/issues/37/mpi4pympiwin-has-the-wrong-size-try

mdjurfeldt commented 8 years ago

Great! Thanks. Den 22 feb 2016 17:02 skrev "Martin Schulze" notifications@github.com:

I found that people had the same error message (with swig or cython wrappers) in the past, so I posted that issue to the mpi4py issue-tracker, here is a link to follow up the current status:

https://bitbucket.org/mpi4py/mpi4py/issues/37/mpi4pympiwin-has-the-wrong-size-try

— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187244791.

uahic commented 8 years ago

Okay the issue is solved, I forked my virtualenv from a module on the Blue Brain VIZ Cluster and in the site-packages there was a mpi4py module. Eventhough it never appeared in the PYTHONPATH it seems that the automake cache in my MUSIC fork was still pointing to that and caused the trouble.

I also updating the python bindings to support Python3 now but new errors with openmpi pop up when executing the examples from pymusic but this is more due to our compiled version than pymusic, anyway....

[bbpviz022.cscs.ch:05596] mca: base: component_find: unable to open /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap: /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so: undefined symbol: opal_show_help (ignored)

Did you ever encounter this?

mdjurfeldt commented 8 years ago

No, but this typically means that the shared library (mca_shmem_mmap.so in this case) can't resolve a symbol, because one of the shared libraries it depends on is not accessible on the library path. You might be able to resolve the problem by providing a correct LD_LIBRARY_PATH.

Do:

ldd /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so

In th output, you'll likely find that libopen-pal.so can't be found. If you do

export LD_LIBRARY_PATH=/gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi:$LD_LIBRARY_PATH http://bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so

and try ldd again, the issue might be resolved.

All of this is likely due to a problem with your openmpi installation and/or module system (if you use that).

Best regards, Mikael

On Tue, Feb 23, 2016 at 6:56 PM, Martin Schulze notifications@github.com wrote:

Okay the issue is solved, I forked my virtualenv from a module on the Blue Brain VIZ Cluster and in the site-packages there was a mpi4py module. Eventhough it never appeared in the PYTHONPATH it seems that the automake cache in my MUSIC fork was still pointing to that and caused the trouble.

I also updating the python bindings to support Python3 now but new errors with openmpi pop up when executing the examples from pymusic

[bbpviz022.cscs.ch:05596] mca: base: component_find: unable to open /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap: /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so: undefined symbol: opal_show_help (ignored)

Did you ever encounter this?

— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187818101.

uahic commented 8 years ago

The ldd does indeed not print libopen-pal.so and the module config is as follows

`/gpfs/bbp.cscs.ch/apps/viz/modulefiles/openmpi/1.8.7-1

module-whatis Loads the openmpi environment for Viz compute nodes

module-whatis man pages: ompi_info mpirun

module-whatis ./configure --enable-mpi-thread-multiple --with-slurm --with-verbs --with-hwloc --with-pmi

module-whatis Full build directory in /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/build

module-whatis IMB benchmark results in /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-default/test.out

module-whatis IMB verification results in /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-DCHECK/test.out

setenv SLURM_MPI_TYPE pmi2

setenv OMPI_HOME /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1

setenv OMPI_WITHOUT_SLURM --mca ess pmi --mca db ^pmi --mca grpcomm ^pmi

setenv OMPI_WITHOUT_RDMA --mca btl openib

setenv OMPI_FORCE_RDMA_THREADING --mca btl_base_thread_multiple_override 1

setenv OMPI_USE_IPOIB --mca btl_tcp_if_include ib0 --mca oob_tcp_if_include ib0

prepend-path MANPATH /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/share/man

prepend-path PATH /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/bin

prepend-path LD_LIBRARY_PATH /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib

conflict mpich conflict mvapich2 conflict openmpi `

After adding the subfolder .../lib/openmpi it still does not work (nor the ldd has changed),

mdjurfeldt commented 8 years ago

If the mca_shmem_mmap.so file was explicitly dependent on libopen-pal.so, it should have listed it, but indicated with question marks that it couldn't find it.

This now instead means that mca_shmem_mmap.so is supposed to be loaded into an image which already has the open-pal library loaded.

It seems like your environment is messed up and you need to discuss this with your sysadmins. On the MUSIC side, you could check that MPI_CXXFLAGS and MPI_LDFLAGS correspond correctly to the output of:

mpicxx -showme:compile

and

mpicxx -shome:link

respectively.

On Tue, Feb 23, 2016 at 7:28 PM, Martin Schulze notifications@github.com wrote:

The ldd does indeed not print libopen-pal.so and the module config is as follows

`/gpfs/bbp.cscs.ch/apps/viz/modulefiles/openmpi/1.8.7-1:

module-whatis Loads the openmpi environment for Viz compute nodes module-whatis man pages: ompi_info mpirun module-whatis ./configure --enable-mpi-thread-multiple --with-slurm --with-verbs --with-hwloc --with-pmi module-whatis Full build directory in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/build module-whatis IMB benchmark results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-default/test.out module-whatis IMB verification results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-DCHECK/test.out setenv SLURM_MPI_TYPE pmi2 setenv OMPI_HOME /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1 setenv OMPI_WITHOUT_SLURM --mca ess ^pmi --mca db ^pmi --mca grpcomm ^pmi setenv OMPI_WITHOUT_RDMA --mca btl ^openib setenv OMPI_FORCE_RDMA_THREADING --mca btl_base_thread_multiple_override 1 setenv OMPI_USE_IPOIB --mca btl_tcp_if_include ib0 --mca oob_tcp_if_include ib0 prepend-path MANPATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/share/man prepend-path PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/bin prepend-path LD_LIBRARY_PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib conflict mpich conflict mvapich2 conflict openmpi `

After adding the subfolder .../lib/openmpi it still does not work (nor the ldd has changed),

— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187830541.

mdjurfeldt commented 8 years ago

Make sure also that you used the correct mpicxx when configuring and compileing MUSIC, i.e. the mpicxx which compiles for the compute nodes.

On Tue, Feb 23, 2016 at 7:49 PM, Mikael Djurfeldt mikael@djurfeldt.com wrote:

If the mca_shmem_mmap.so file was explicitly dependent on libopen-pal.so, it should have listed it, but indicated with question marks that it couldn't find it.

This now instead means that mca_shmem_mmap.so is supposed to be loaded into an image which already has the open-pal library loaded.

It seems like your environment is messed up and you need to discuss this with your sysadmins. On the MUSIC side, you could check that MPI_CXXFLAGS and MPI_LDFLAGS correspond correctly to the output of:

mpicxx -showme:compile

and

mpicxx -shome:link

respectively.

On Tue, Feb 23, 2016 at 7:28 PM, Martin Schulze notifications@github.com wrote:

The ldd does indeed not print libopen-pal.so and the module config is as follows

`/gpfs/bbp.cscs.ch/apps/viz/modulefiles/openmpi/1.8.7-1:

module-whatis Loads the openmpi environment for Viz compute nodes module-whatis man pages: ompi_info mpirun module-whatis ./configure --enable-mpi-thread-multiple --with-slurm --with-verbs --with-hwloc --with-pmi module-whatis Full build directory in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/build module-whatis IMB benchmark results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-default/test.out module-whatis IMB verification results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-DCHECK/test.out setenv SLURM_MPI_TYPE pmi2 setenv OMPI_HOME /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1 setenv OMPI_WITHOUT_SLURM --mca ess ^pmi --mca db ^pmi --mca grpcomm ^pmi setenv OMPI_WITHOUT_RDMA --mca btl ^openib setenv OMPI_FORCE_RDMA_THREADING --mca btl_base_thread_multiple_override 1 setenv OMPI_USE_IPOIB --mca btl_tcp_if_include ib0 --mca oob_tcp_if_include ib0 prepend-path MANPATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/share/man prepend-path PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/bin prepend-path LD_LIBRARY_PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib conflict mpich conflict mvapich2 conflict openmpi `

After adding the subfolder .../lib/openmpi it still does not work (nor the ldd has changed),

— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187830541.

uahic commented 8 years ago

hmh the mpicxx was correct, after retrying several times I decided to build my own openmpi which also suffered from the same issue; then I found this site helpful as the flag --disable-dlopen helped and in short there is an explanation why. Maybe the reference how the mpi4py worked around that issue is helpful for music too (could it be that only openmpi has this issue?)

mdjurfeldt commented 8 years ago

Good that you got it working.

Which site did you find? Den 24 feb 2016 02:08 skrev "Martin Schulze" notifications@github.com:

hmh the mpicxx was correct, after retrying several times I decided to build my own openmpi which also suffered from the same issue; then I found this site helpful as the flag --disable-dlopen helped and in short there is an explanation why, finally its running. Maybe the reference how the mpi4py worked around that issue is helpful for music too (could be that only openmpi has this issue?)

— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187996270.

uahic commented 8 years ago

You can find it at the section "Notes" (bottom of the page), https://github.com/jhedev/mpi_python

mdjurfeldt commented 8 years ago

Thanks.

It seems to me that this issue is not directly related to MUSIC. However, we could document the problem and give advice to users.

I'll leave this issue open until that is done.

On Sat, Feb 27, 2016 at 4:42 PM, Martin Schulze notifications@github.com wrote:

You can find it at the section "Notes" (bottom of the page), https://github.com/jhedev/mpi_python

— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-189669711.