Closed sebwolf-de closed 1 year ago
To run the training on Frontera, you have to follow the instructions in singularity.md
.
TPV13 took 38s to complete :D Visualization in the notebook does not work, but paraview is already installed on Frontera ;-)
Actually, instead of building a different container for singularity, a better solution is to detect whether it is a docker or singularity environment in the entrypoint script. Basically, you can have something like
if grep "docker/containers" /proc/self/mountinfo -qa; then
# the script is in docker, do nothing special
else
# it is not in docker, thus singularity
cp -r /home/training ~/.seissol-training
# something else
fi
I have used this in my mspass container, and it works pretty reliable.
btw, I would prefer using pwd
instead of ~/
because of the quota limit in the home directory. On TACC's system, $HOME
, $WORK
and $SCRATCH
are by default mounted to the container, so it is guaranteed that pwd
will always work. I guess the caveat is that for other systems that doesn't have such default, it may fail. Well, I guess ~/
can also fail if home is not mounted by default.
@wangyinz good ideas: I didn't have too much experience with singularity, so that's the workaround, I've found. Feel free to add your suggestions.
#!/bin/bash
set -o errexit
ulimit -Ss unlimited
if grep "docker/containers" /proc/self/mountinfo -qa; then
cd /home/training
else
# it is not in docker, thus singularity
cp -r /home/training ${PWD}/seissol-training
cd ${PWD}/seissol-training
fi
if [ -z $1 ]
then
set -x
export DISPLAY=:99.0
export PYVISTA_OFF_SCREEN=true
export PYVISTA_USE_IPYVTK=true
which Xvfb
Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
sleep 3
set +x
jupyter lab --allow-root --port=53155 --no-browser --ip=0.0.0.0
exit
fi
case "$1" in
gmsh|pumgen|rconv)
exec "$@"
;;
seissol)
set -- SeisSol_Release_dhsw_4_elastic "${@:2}"
exec "$@"
;;
seissol_viscoelastic)
set -- SeisSol_Release_dhsw_4_viscoelastic2 "${@:2}"
exec "$@"
;;
*)
exec "$@"
;;
esac
Basically, I would change the content of entrypoint.sh
to above. Then, it should run fine on both docker and singularity (and probably other container runtimes on HPC). If you can give me access to this repo, I can update that file directly.
p.s. this is untested, but you get the point.
I've given you access. Then you can also try it, once the container is pushed on dockerhub ;-)
Thanks! I updated the script already. Will test it out soon.
edited: Tested on Frontera, and it runs fine.
Do you mind if I also add the SCOPED containerized badge to this repo?
We do this to the containers supported by the SCOPED base container.
Two things:
~/seissol-training/training
Good point! Thanks for fixing that. To run MPI on Frontera, it is required to have the mpi launcher outside of the container. Below is the example that I sent Thomas a while ago:
c204-013[clx](505)$ ml tacc-apptainer
c204-013[clx](508)$ singularity pull docker://wangyinz/seissoltraining:test
INFO: Using cached SIF image
c204-013[clx](510)$ pwd
/scratch1/06058/iwang/training/seissol
c204-013[clx](511)$ singularity exec seissoltraining_test.sif cp -r /home/training/sulawesi ./
c204-013[clx](512)$ ls
seissoltraining_test.sif sulawesi
c204-013[clx](513)$ cd sulawesi/
c204-013[clx](514)$ mpirun -n 2 singularity run ../seissoltraining_test.sif seissol parametersLSW.par
Thu May 04 15:03:31, Info: Welcome to SeisSol
Thu May 04 15:03:31, Info: Copyright (c) 2012-2021, SeisSol Group
Thu May 04 15:03:31, Info: Built on: Apr 6 2023 18:06:06
Thu May 04 15:03:31, Info: Version: 202103_Sumatra-1126-g973cc45b
Thu May 04 15:03:31, Info: Running on: c204-013.frontera.tacc.utexas.edu
Thu May 04 15:03:31, Info: Using MPI with #ranks: 2
Thu May 04 15:03:31, Info: Using OMP with #threads/rank: 1
Thu May 04 15:03:31, Info: OpenMP worker affinity (this process): "0---------|----------|----------|----------|----------|------"
Thu May 04 15:03:31, Info: OpenMP worker affinity (this node) : "01--------|----------|----------|----------|----------|------"
Thu May 04 15:03:31, Info: The stack size ulimit is unlimited.
Thu May 04 15:03:31, Info: Input Parameters:
0x55d835c7c880
Rank: 0 | Info | <--------------------------------------------------------->
Rank: 0 | Info | < SeisSol MPI initialization >
Rank: 0 | Info | <--------------------------------------------------------->
Rank: 0 | Info | Double precision used for real.
Rank: 0 | Info | <--------------------------------------------------------->
INFORMATION: The assumed unit number is 6 for stdout and 0 for stderr.
If no information follows, please change the value.
Rank: 0 | Info | <--------------------------------------------------------->
Rank: 0 | Info | < Start ini_SeisSol ... >
Rank: 0 | Info | <--------------------------------------------------------->
Rank: 0 | Info | < Parameters read from file: parametersLSW.par >
Rank: 0 | Info | < >
Rank: 0 | Info | (Drucker-Prager) plasticity assumed .
Rank: 0 | Info | Plastic relaxation Tv is set to: 5.0000000000000003E-002
Rank: 0 | Info | No attenuation assumed.
Rank: 0 | Info | No adjoint wavefield generated.
Rank: 0 | Info | Isotropic material is assumed.
Rank: 0 | Info | Read a PUML mesh file
Rank: 0 | Warning | Ignoring space order from parameter file, using 4
Rank: 0 | Info | Volume output is disabled
Rank: 0 | Info | Use HDF5 XdmfWriter backend
Basically, the way that container works is that the MPI library outside of the container is ABI compatible with the one inside the container, so that the MPI program can properly communicate with the interconnect through MPI library. On HPC, the container is not a fully isolated environment - we need access to the device for inter-node communication.
That's also why it hangs when launching MPI launcher within the container. It cannot use the device properly.
Thanks. I didn't mean that way, but from inside the jupyter lab.
I see... Yeah, that will not work inside the container
Well, I mean it will run fine launching the mpirun from outside the container. But yes, in jupyter notebook, this will not work
Hmm, that used to work last week :/
hmmm... are you sure? Let me give it a try then. I don't think we changed anything that would break it
Ah, last week, I tried seissol/training:latest
and not seissol/training:pr-33
Aha... that makes sense. The latest
has openMPI built-in, and it is not compatible with the device on Frontera, so it does not have the same issue.
I just tried from the outside of the container, and it works as expected:
$ ibrun -np 1 apptainer run ../../training_pr-33.sif pumgen -s msh2 tpv13_training.msh
TACC: Starting up job 5447847
TACC: Starting parallel tasks...
Mon May 08 15:33:38, Info: Using GMSH mesh format 2 (msh2) mesh
Mon May 08 15:33:38, Info: Create APF connectivity
Mon May 08 15:33:38, Info: Set coordinates in APF
Mon May 08 15:33:38, Info: Mesh size: 37266
Mon May 08 15:33:38, Info: Minimum insphere found: 54.6127
Mon May 08 15:33:38, Info: Writing cells
Mon May 08 15:33:38, Info: Writing vertices
Mon May 08 15:33:38, Info: Writing group information
Mon May 08 15:33:38, Info: Writing boundary condition
Mon May 08 15:33:38, Info: Writing XDMF file
Mon May 08 15:33:38, Info: Finished successfully
TACC: Shutdown complete. Exiting.
Performance on a single node is ~400 GFLOP/s, not too bad, but also not great. Maybe also because the meshes are so small.
Okay, it's an hsw
order 4 build in the container, so performance is as good as it gets IMHO.
Cool! After this training done, we can actually build a definitive version with skx and order 6, and it will run at large scale on Frontera (and Frontera alike systems). This can be something to be added to the main seissol repo.
Great!!!!!!!!!!!!!!
On Mon, May 8, 2023 at 8:53 PM Ian Wang @.***> wrote:
Cool! After this training done, we can actually build a definitive version with skx and order 6, and it will run at large scale on Frontera (and Frontera alike systems). This can be something to be added to the main seissol repo.
— Reply to this email directly, view it on GitHub https://github.com/SeisSol/Training/pull/33#issuecomment-1539092580, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHOTRFWGSZMVA2OAK3JNRTXFG5UJANCNFSM6AAAAAAXW5GL4I . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Mostly a copy of #32, but with newer dependencies.
Check all other scenarios on Frontera