Open uahic opened 8 years ago
I expect that mpi4py was compiled with a different mpi than than music -- does your environment have a module system? You may need to do a module load xxxx before building and/or running music.
For the .pymusic --- yes, pymusic needs to be updated to python3+
I was not sure if I really compiled everything with the same mpi module (yep we do have the module system on the cluster) but now I recompiled everything and made sure that the make-scripts of music and mpi4py take the same openmpi-1.8.7 package. The issue still exists, so probably I need to have a look in the music implementation and if the problem is not there to ask the mpi4py guy. Mpi4py seems to work so far without any complaints
I found that people had the same error message (with swig or cython wrappers) in the past, so I posted that issue to the mpi4py issue-tracker, here is a link to follow up the current status:
https://bitbucket.org/mpi4py/mpi4py/issues/37/mpi4pympiwin-has-the-wrong-size-try
Great! Thanks. Den 22 feb 2016 17:02 skrev "Martin Schulze" notifications@github.com:
I found that people had the same error message (with swig or cython wrappers) in the past, so I posted that issue to the mpi4py issue-tracker, here is a link to follow up the current status:
https://bitbucket.org/mpi4py/mpi4py/issues/37/mpi4pympiwin-has-the-wrong-size-try
— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187244791.
Okay the issue is solved, I forked my virtualenv from a module on the Blue Brain VIZ Cluster and in the site-packages there was a mpi4py module. Eventhough it never appeared in the PYTHONPATH it seems that the automake cache in my MUSIC fork was still pointing to that and caused the trouble.
I also updating the python bindings to support Python3 now but new errors with openmpi pop up when executing the examples from pymusic but this is more due to our compiled version than pymusic, anyway....
[bbpviz022.cscs.ch:05596] mca: base: component_find: unable to open /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap: /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so: undefined symbol: opal_show_help (ignored)
Did you ever encounter this?
No, but this typically means that the shared library (mca_shmem_mmap.so in this case) can't resolve a symbol, because one of the shared libraries it depends on is not accessible on the library path. You might be able to resolve the problem by providing a correct LD_LIBRARY_PATH.
Do:
ldd /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so
In th output, you'll likely find that libopen-pal.so can't be found. If you do
export LD_LIBRARY_PATH=/gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi:$LD_LIBRARY_PATH http://bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so
and try ldd again, the issue might be resolved.
All of this is likely due to a problem with your openmpi installation and/or module system (if you use that).
Best regards, Mikael
On Tue, Feb 23, 2016 at 6:56 PM, Martin Schulze notifications@github.com wrote:
Okay the issue is solved, I forked my virtualenv from a module on the Blue Brain VIZ Cluster and in the site-packages there was a mpi4py module. Eventhough it never appeared in the PYTHONPATH it seems that the automake cache in my MUSIC fork was still pointing to that and caused the trouble.
I also updating the python bindings to support Python3 now but new errors with openmpi pop up when executing the examples from pymusic
[bbpviz022.cscs.ch:05596] mca: base: component_find: unable to open /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap: /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib/openmpi/mca_shmem_mmap.so: undefined symbol: opal_show_help (ignored)
Did you ever encounter this?
— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187818101.
The ldd does indeed not print libopen-pal.so and the module config is as follows
`/gpfs/bbp.cscs.ch/apps/viz/modulefiles/openmpi/1.8.7-1
module-whatis Loads the openmpi environment for Viz compute nodes
module-whatis man pages: ompi_info mpirun
module-whatis ./configure --enable-mpi-thread-multiple --with-slurm --with-verbs --with-hwloc --with-pmi
module-whatis Full build directory in /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/build
module-whatis IMB benchmark results in /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-default/test.out
module-whatis IMB verification results in /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-DCHECK/test.out
setenv SLURM_MPI_TYPE pmi2
setenv OMPI_HOME /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1
setenv OMPI_WITHOUT_SLURM --mca ess pmi --mca db ^pmi --mca grpcomm ^pmi
setenv OMPI_WITHOUT_RDMA --mca btl openib
setenv OMPI_FORCE_RDMA_THREADING --mca btl_base_thread_multiple_override 1
setenv OMPI_USE_IPOIB --mca btl_tcp_if_include ib0 --mca oob_tcp_if_include ib0
prepend-path MANPATH /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/share/man
prepend-path PATH /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/bin
prepend-path LD_LIBRARY_PATH /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib
conflict mpich conflict mvapich2 conflict openmpi `
After adding the subfolder .../lib/openmpi it still does not work (nor the ldd has changed),
If the mca_shmem_mmap.so file was explicitly dependent on libopen-pal.so, it should have listed it, but indicated with question marks that it couldn't find it.
This now instead means that mca_shmem_mmap.so is supposed to be loaded into an image which already has the open-pal library loaded.
It seems like your environment is messed up and you need to discuss this with your sysadmins. On the MUSIC side, you could check that MPI_CXXFLAGS and MPI_LDFLAGS correspond correctly to the output of:
mpicxx -showme:compile
and
mpicxx -shome:link
respectively.
On Tue, Feb 23, 2016 at 7:28 PM, Martin Schulze notifications@github.com wrote:
The ldd does indeed not print libopen-pal.so and the module config is as follows
`/gpfs/bbp.cscs.ch/apps/viz/modulefiles/openmpi/1.8.7-1:
module-whatis Loads the openmpi environment for Viz compute nodes module-whatis man pages: ompi_info mpirun module-whatis ./configure --enable-mpi-thread-multiple --with-slurm --with-verbs --with-hwloc --with-pmi module-whatis Full build directory in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/build module-whatis IMB benchmark results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-default/test.out module-whatis IMB verification results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-DCHECK/test.out setenv SLURM_MPI_TYPE pmi2 setenv OMPI_HOME /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1 setenv OMPI_WITHOUT_SLURM --mca ess ^pmi --mca db ^pmi --mca grpcomm ^pmi setenv OMPI_WITHOUT_RDMA --mca btl ^openib setenv OMPI_FORCE_RDMA_THREADING --mca btl_base_thread_multiple_override 1 setenv OMPI_USE_IPOIB --mca btl_tcp_if_include ib0 --mca oob_tcp_if_include ib0 prepend-path MANPATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/share/man prepend-path PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/bin prepend-path LD_LIBRARY_PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib conflict mpich conflict mvapich2 conflict openmpi `
After adding the subfolder .../lib/openmpi it still does not work (nor the ldd has changed),
— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187830541.
Make sure also that you used the correct mpicxx when configuring and compileing MUSIC, i.e. the mpicxx which compiles for the compute nodes.
On Tue, Feb 23, 2016 at 7:49 PM, Mikael Djurfeldt mikael@djurfeldt.com wrote:
If the mca_shmem_mmap.so file was explicitly dependent on libopen-pal.so, it should have listed it, but indicated with question marks that it couldn't find it.
This now instead means that mca_shmem_mmap.so is supposed to be loaded into an image which already has the open-pal library loaded.
It seems like your environment is messed up and you need to discuss this with your sysadmins. On the MUSIC side, you could check that MPI_CXXFLAGS and MPI_LDFLAGS correspond correctly to the output of:
mpicxx -showme:compile
and
mpicxx -shome:link
respectively.
On Tue, Feb 23, 2016 at 7:28 PM, Martin Schulze notifications@github.com wrote:
The ldd does indeed not print libopen-pal.so and the module config is as follows
`/gpfs/bbp.cscs.ch/apps/viz/modulefiles/openmpi/1.8.7-1:
module-whatis Loads the openmpi environment for Viz compute nodes module-whatis man pages: ompi_info mpirun module-whatis ./configure --enable-mpi-thread-multiple --with-slurm --with-verbs --with-hwloc --with-pmi module-whatis Full build directory in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/build module-whatis IMB benchmark results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-default/test.out module-whatis IMB verification results in /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/imb-DCHECK/test.out setenv SLURM_MPI_TYPE pmi2 setenv OMPI_HOME /gpfs/bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1 setenv OMPI_WITHOUT_SLURM --mca ess ^pmi --mca db ^pmi --mca grpcomm ^pmi setenv OMPI_WITHOUT_RDMA --mca btl ^openib setenv OMPI_FORCE_RDMA_THREADING --mca btl_base_thread_multiple_override 1 setenv OMPI_USE_IPOIB --mca btl_tcp_if_include ib0 --mca oob_tcp_if_include ib0 prepend-path MANPATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/share/man prepend-path PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/bin prepend-path LD_LIBRARY_PATH /gpfs/ bbp.cscs.ch/apps/viz/tools/openmpi/openmpi-1.8.7-1/install/lib conflict mpich conflict mvapich2 conflict openmpi `
After adding the subfolder .../lib/openmpi it still does not work (nor the ldd has changed),
— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187830541.
hmh the mpicxx was correct, after retrying several times I decided to build my own openmpi which also suffered from the same issue; then I found this site helpful as the flag --disable-dlopen helped and in short there is an explanation why. Maybe the reference how the mpi4py worked around that issue is helpful for music too (could it be that only openmpi has this issue?)
Good that you got it working.
Which site did you find? Den 24 feb 2016 02:08 skrev "Martin Schulze" notifications@github.com:
hmh the mpicxx was correct, after retrying several times I decided to build my own openmpi which also suffered from the same issue; then I found this site helpful as the flag --disable-dlopen helped and in short there is an explanation why, finally its running. Maybe the reference how the mpi4py worked around that issue is helpful for music too (could be that only openmpi has this issue?)
— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-187996270.
You can find it at the section "Notes" (bottom of the page), https://github.com/jhedev/mpi_python
Thanks.
It seems to me that this issue is not directly related to MUSIC. However, we could document the problem and give advice to users.
I'll leave this issue open until that is done.
On Sat, Feb 27, 2016 at 4:42 PM, Martin Schulze notifications@github.com wrote:
You can find it at the section "Notes" (bottom of the page), https://github.com/jhedev/mpi_python
— Reply to this email directly or view it on GitHub https://github.com/INCF/MUSIC/issues/10#issuecomment-189669711.
I've compiled the recent music version and tried to import it with Ipython, the first error Ive got is that when it enteres the init.py of the music package in the build path, it cannot find pymusic as it is imported 'absolute'. So I just modified the line to from .pymusic import *
After that another issue arises where I am not sure if I am using the package in a wrong manner. Ive tried it also to use with mpiexec ipython on a cluster (openmpi 1.8.7, mpi4py 2.0.0). The python interpreter is from a virtual environment for python3.4 (freshly setup with Cython and mpi4py), MUSIC ENV's are set as follows:
export MUSIC_ROOT_DIR=/gpfs/bbp.cscs.ch/home/bbpnrsoa/martin.setup/builds/MUSIC_1_1_15 export MUSIC_ROOT=${MUSIC_ROOT_DIR} MUSIC_PATH=${MUSIC_ROOT_DIR} export LD_LIBRARY_PATH=${MUSIC_PATH}/lib:$LD_LIBRARY_PATH export PATH=${MUSIC_PATH}/bin:$PATH export CPATH=${MUSIC_PATH}/include:$CPATH export PYTHONPATH=${MUSIC_PATH}/lib/python3.4/site-packages:$PYTHONPATH
` iPython 3.4.3 (default, Jan 18 2016, 16:48:08) Type "copyright", "credits" or "license" for more information.
IPython 4.1.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.
import music
ValueError Traceback (most recent call last)