Sebastian/singularity

[sebwolf-de]

Mostly a copy of #32, but with newer dependencies.

• [x] Check TPV13 locally
• [x] Check TPV13 on Frontera
• [x] Check performance on Frontera

Check all other scenarios on Frontera

• [x] Kaikoura LSW
• [x] Kaikoura RS
• [x] Northridge
• [x] Sulawesi LSW
• [x] Sulawesi RS

[sebwolf-de]

To run the training on Frontera, you have to follow the instructions in singularity.md. TPV13 took 38s to complete :D Visualization in the notebook does not work, but paraview is already installed on Frontera ;-)

[wangyinz]

Actually, instead of building a different container for singularity, a better solution is to detect whether it is a docker or singularity environment in the entrypoint script. Basically, you can have something like

if grep "docker/containers" /proc/self/mountinfo -qa; then
  # the script is in docker, do nothing special
else
  # it is not in docker, thus singularity
  cp -r /home/training ~/.seissol-training
  # something else
fi

I have used this in my mspass container, and it works pretty reliable.

[wangyinz]

btw, I would prefer using pwd instead of ~/ because of the quota limit in the home directory. On TACC's system, $HOME, $WORK and $SCRATCH are by default mounted to the container, so it is guaranteed that pwd will always work. I guess the caveat is that for other systems that doesn't have such default, it may fail. Well, I guess ~/ can also fail if home is not mounted by default.

[sebwolf-de]

@wangyinz good ideas: I didn't have too much experience with singularity, so that's the workaround, I've found. Feel free to add your suggestions.

[wangyinz]

#!/bin/bash
set -o errexit
ulimit -Ss unlimited

if grep "docker/containers" /proc/self/mountinfo -qa; then
    cd /home/training
else
    # it is not in docker, thus singularity
    cp -r /home/training ${PWD}/seissol-training
    cd ${PWD}/seissol-training
fi

if [ -z $1 ]
then
    set -x
    export DISPLAY=:99.0
    export PYVISTA_OFF_SCREEN=true
    export PYVISTA_USE_IPYVTK=true
    which Xvfb
    Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
    sleep 3
    set +x
    jupyter lab --allow-root --port=53155 --no-browser --ip=0.0.0.0
    exit
fi

case "$1" in
    gmsh|pumgen|rconv)
        exec "$@"
    ;;
    seissol)
        set -- SeisSol_Release_dhsw_4_elastic "${@:2}"
        exec "$@"
    ;;
    seissol_viscoelastic)
        set -- SeisSol_Release_dhsw_4_viscoelastic2 "${@:2}"
        exec "$@"
    ;;
    *)
        exec "$@"
    ;;
esac

Basically, I would change the content of entrypoint.sh to above. Then, it should run fine on both docker and singularity (and probably other container runtimes on HPC). If you can give me access to this repo, I can update that file directly.

p.s. this is untested, but you get the point.

[sebwolf-de]

I've given you access. Then you can also try it, once the container is pushed on dockerhub ;-)

[wangyinz]

Thanks! I updated the script already. Will test it out soon.

edited: Tested on Frontera, and it runs fine.

[wangyinz]

Do you mind if I also add the SCOPED containerized badge to this repo?

We do this to the containers supported by the SCOPED base container.

[sebwolf-de]

Two things:

• If I execute the container twice from my home directory, it will create ~/seissol-training/training
• I can't run MPI programs in the container on Frontera.

[wangyinz]

Good point! Thanks for fixing that. To run MPI on Frontera, it is required to have the mpi launcher outside of the container. Below is the example that I sent Thomas a while ago:

c204-013[clx](505)$ ml tacc-apptainer
c204-013[clx](508)$ singularity pull docker://wangyinz/seissoltraining:test
INFO:    Using cached SIF image
c204-013[clx](510)$ pwd
/scratch1/06058/iwang/training/seissol
c204-013[clx](511)$ singularity exec seissoltraining_test.sif cp -r /home/training/sulawesi ./
c204-013[clx](512)$ ls
seissoltraining_test.sif  sulawesi
c204-013[clx](513)$ cd sulawesi/
c204-013[clx](514)$ mpirun -n 2 singularity run ../seissoltraining_test.sif seissol parametersLSW.par
Thu May 04 15:03:31, Info:  Welcome to SeisSol
Thu May 04 15:03:31, Info:  Copyright (c) 2012-2021, SeisSol Group
Thu May 04 15:03:31, Info:  Built on: Apr  6 2023 18:06:06
Thu May 04 15:03:31, Info:  Version: 202103_Sumatra-1126-g973cc45b
Thu May 04 15:03:31, Info:  Running on: c204-013.frontera.tacc.utexas.edu
Thu May 04 15:03:31, Info:  Using MPI with #ranks: 2
Thu May 04 15:03:31, Info:  Using OMP with #threads/rank: 1
Thu May 04 15:03:31, Info:  OpenMP worker affinity (this process): "0---------|----------|----------|----------|----------|------"
Thu May 04 15:03:31, Info:  OpenMP worker affinity (this node)   : "01--------|----------|----------|----------|----------|------"
Thu May 04 15:03:31, Info:  The stack size ulimit is unlimited.
Thu May 04 15:03:31, Info:  Input Parameters:
 0x55d835c7c880
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <                SeisSol MPI initialization               >
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    |  Double precision used for real.
Rank:        0 | Info    | <--------------------------------------------------------->
 INFORMATION: The assumed unit number is           6 for stdout and           0 for stderr.
              If no information follows, please change the value.
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <     Start ini_SeisSol ...                               >
Rank:        0 | Info    | <--------------------------------------------------------->
Rank:        0 | Info    | <  Parameters read from file: parametersLSW.par              >
Rank:        0 | Info    | <                                                         >
Rank:        0 | Info    | (Drucker-Prager) plasticity assumed .
Rank:        0 | Info    | Plastic relaxation Tv is set to:    5.0000000000000003E-002
Rank:        0 | Info    | No attenuation assumed.
Rank:        0 | Info    | No adjoint wavefield generated.
Rank:        0 | Info    | Isotropic material is assumed.
Rank:        0 | Info    | Read a PUML mesh file
Rank:        0 | Warning | Ignoring space order from parameter file, using           4
Rank:        0 | Info    | Volume output is disabled
Rank:        0 | Info    | Use HDF5 XdmfWriter backend

Basically, the way that container works is that the MPI library outside of the container is ABI compatible with the one inside the container, so that the MPI program can properly communicate with the interconnect through MPI library. On HPC, the container is not a fully isolated environment - we need access to the device for inter-node communication.

That's also why it hangs when launching MPI launcher within the container. It cannot use the device properly.

[sebwolf-de]

Thanks. I didn't mean that way, but from inside the jupyter lab.

[wangyinz]

I see... Yeah, that will not work inside the container

Well, I mean it will run fine launching the mpirun from outside the container. But yes, in jupyter notebook, this will not work

[sebwolf-de]

Hmm, that used to work last week :/

[wangyinz]

hmmm... are you sure? Let me give it a try then. I don't think we changed anything that would break it

[sebwolf-de]

Ah, last week, I tried seissol/training:latest and not seissol/training:pr-33

[wangyinz]

Aha... that makes sense. The latest has openMPI built-in, and it is not compatible with the device on Frontera, so it does not have the same issue.

I just tried from the outside of the container, and it works as expected:

$ ibrun -np 1 apptainer run ../../training_pr-33.sif pumgen -s msh2 tpv13_training.msh
TACC:  Starting up job 5447847 
TACC:  Starting parallel tasks... 
Mon May 08 15:33:38, Info:  Using GMSH mesh format 2 (msh2) mesh 
Mon May 08 15:33:38, Info:  Create APF connectivity 
Mon May 08 15:33:38, Info:  Set coordinates in APF 
Mon May 08 15:33:38, Info:  Mesh size: 37266 
Mon May 08 15:33:38, Info:  Minimum insphere found: 54.6127 
Mon May 08 15:33:38, Info:  Writing cells 
Mon May 08 15:33:38, Info:  Writing vertices 
Mon May 08 15:33:38, Info:  Writing group information 
Mon May 08 15:33:38, Info:  Writing boundary condition 
Mon May 08 15:33:38, Info:  Writing XDMF file 
Mon May 08 15:33:38, Info:  Finished successfully 
TACC:  Shutdown complete. Exiting. 

[sebwolf-de]

Performance on a single node is ~400 GFLOP/s, not too bad, but also not great. Maybe also because the meshes are so small.

[sebwolf-de]

Okay, it's an hsw order 4 build in the container, so performance is as good as it gets IMHO.

[wangyinz]

Cool! After this training done, we can actually build a definitive version with skx and order 6, and it will run at large scale on Frontera (and Frontera alike systems). This can be something to be added to the main seissol repo.

[AliceGabriel]

Great!!!!!!!!!!!!!!

On Mon, May 8, 2023 at 8:53 PM Ian Wang @.***> wrote:

Cool! After this training done, we can actually build a definitive version with skx and order 6, and it will run at large scale on Frontera (and Frontera alike systems). This can be something to be added to the main seissol repo.

— Reply to this email directly, view it on GitHub https://github.com/SeisSol/Training/pull/33#issuecomment-1539092580, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHOTRFWGSZMVA2OAK3JNRTXFG5UJANCNFSM6AAAAAAXW5GL4I . You are receiving this because you are subscribed to this thread.Message ID: @.***>