search

NanoComp / meep

free finite-difference time-domain (FDTD) software for electromagnetic simulations
GNU General Public License v2.0
1.22k stars 621 forks source link

Jupyterhub meep crash #1411

Open hoba87 opened 4 years ago

hoba87 commented 4 years ago

I'm using jupyter with jupyterhub and when I try to import meep the kernel is crashing during the MPI init:

*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)

If I use python or ipython from the console or even start a jupyter notebook directly from the console the import of meep is no problem. The environment variables PATH and LD_LIBRARY_PATH are nearly the same. Jupyterhub:

PATH=/opt/openmpi/openmpi-4.0.5/bin:/opt/pmix/pmix-3.1.5/bin:/opt/python/python-3.7.9/bin:/opt/hdf5/hdf5-1.10.7/bin:/opt/libctl/libctl-4.5.0/bin:/opt/gmsh/gmsh-4.6.0/bin:/opt/h5utils/h5utils-1.13.1/bin:/opt/ffmpeg/ffmpeg-4.3.1/bin:/opt/netcdf/netcdf-c-4.7.4-f-4.5.3/bin:/opt/harminv/harminv-1.4.1/bin:/opt/mpb/mpb-1.10.0/bin:/usr/share/Modules/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/slurm/slurm-20.02.5/bin:
LD_LIBRARY_PATH=/opt/openmpi/openmpi-4.0.5/lib:/opt/pmix/pmix-3.1.5/lib:/opt/python/python-3.7.9/lib:/usr/lib64:/opt/hdf5/hdf5-1.10.7/lib:/opt/libctl/libctl-4.5.0/lib:/opt/openblas/openblas-0.3.10/lib:/opt/fftw/fftw-3.3.8/lib:/opt/harminv/harminv-1.4.1/lib:/opt/nlopt/nlopt-2.6.2/lib64:/opt/gsl/gsl-2.6/lib:/opt/libgdsii/libgdsii-0.21/lib:/opt/mpb/mpb-1.10.0/lib:/opt/opencascade/opencascade-7.4.0/lib:/opt/ffmpeg/ffmpeg-4.3.1/lib:/opt/netcdf/netcdf-c-4.7.4-f-4.5.3/lib:/opt/scalapack/scalapack-2.1.0/lib:/opt/pnfft/pnfft-1.0.7-alpha/lib:/opt/pfft/pfft-1.0.8-alpha/lib:/opt/libvdwxc/libvdwxc-0.4.0/lib:/opt/libxc/libxc-4.3.4/lib:

console, etc:

PATH=/opt/openmpi/openmpi-4.0.5/bin:/opt/pmix/pmix-3.1.5/bin:/opt/hdf5/hdf5-1.10.7/bin:/opt/libctl/libctl-4.5.0/bin:/opt/gmsh/gmsh-4.6.0/bin:/opt/h5utils/h5utils-1.13.1/bin:/opt/ffmpeg/ffmpeg-4.3.1/bin:/opt/netcdf/netcdf-c-4.7.4-f-4.5.3/bin:/opt/harminv/harminv-1.4.1/bin:/opt/mpb/mpb-1.10.0/bin:/opt/python/python-3.7.9/bin:/usr/share/Modules/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/slurm/slurm-20.02.5/bin:
LD_LIBRARY_PATH=/opt/openmpi/openmpi-4.0.5/lib:/opt/pmix/pmix-3.1.5/lib:/opt/hdf5/hdf5-1.10.7/lib:/opt/libctl/libctl-4.5.0/lib:/opt/openblas/openblas-0.3.10/lib:/opt/fftw/fftw-3.3.8/lib:/opt/harminv/harminv-1.4.1/lib:/opt/nlopt/nlopt-2.6.2/lib64:/opt/gsl/gsl-2.6/lib:/opt/libgdsii/libgdsii-0.21/lib:/opt/mpb/mpb-1.10.0/lib:/opt/opencascade/opencascade-7.4.0/lib:/opt/ffmpeg/ffmpeg-4.3.1/lib:/opt/netcdf/netcdf-c-4.7.4-f-4.5.3/lib:/opt/scalapack/scalapack-2.1.0/lib:/opt/pnfft/pnfft-1.0.7-alpha/lib:/opt/pfft/pfft-1.0.8-alpha/lib:/opt/libvdwxc/libvdwxc-0.4.0/lib:/opt/libxc/libxc-4.3.4/lib:/opt/python/python-3.7.9/lib:/usr/lib64:

Any idea why this could happen?

stevengj commented 3 years ago

No idea, sorry. Like you, my first suspicion is that some environment variable is different — not that you can set environment variables for notebooks in the Jupyter kernelspec file so maybe you could experiment with that.

hoba87 commented 3 years ago

Just out of curiosity I have tried again to import meep, and now it works also for jupyter with jupyterhub. I don't exactly remember what I have changed since I had this issue, but perhaps it is the env_keep in the jupyterhub_config.py:

c.Spawner.env_keep = ['PATH', 'LD_LIBRARY_PATH', 'PYTHONPATH', 'VIRTUAL_ENV', 'LANG', 'LC_ALL', 'MKL_NUM_THREADS', 'OMP_NUM_THREADS', 'OPENBLAS_NUM_THREADS', 'BASH_FUNC__moduleraw', 'BASH_FUNC_switchml', 'BASH_FUNC_module', 'MODULESHOME', 'MODULEPATH', 'MODULES_CMD']