Open SomePersonSomeWhereInTheWorld opened 1 year ago
Hi @RobbieTheK, sorry for being slow to respond. Yes, both issues seem to be related to the interaction between the images and the system you are running the images on. In order to say more I would need to know more about the system you are using, but here are some general information about the images:
geodynamics/aspect:latest-tacc
looks like an MPI error. It looks like you are not using one of the TACC systems to run this, is this correct? In this case the specialized MPI inside this container is likely just not working with your system (e.g. it might try to communicate in a way that is not supported by your cluster).geodynamics/aspect:latest
uses an unmodified OpenMPI that should work on most normal systems. It is not optimized for Infiniband or other high-speed interconnects. The error you get in the second part of your message looks like you are running this on some cluster that uses SLURM? Are you sure that you have set up your job script correctly that you can start a program successfully? Try if you can run something like echo
in parallel, e.g. as in mpirun -np 4 echo hello
, if this does not succeed (i.e. you dont see 4 output lines displaying hello
) then your problem is not related to ASPECT or its docker image, but to how you set up your job on the cluster.Re: 1st error, correct not a TACC system was just trying to see if it would work.
This is a Bright Computing 9.1 cluster running RHEL 8 with Slurm 20, openmpi/gcc/64/4.1.5a1
I'm using an interactive srun job -c4 -n4 as options.
mpirun -np 4 echo hello hello
hello
hello
hello
Same error:
singularity run aspect.sif aspect-release slab_detachment.prm
[g225:3682368] OPAL ERROR: Unreachable in file ext3x_client.c at line 112
--------------------------------------------------------------------------
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:
version 16.05 or later: you can use SLURM's PMIx support. This
requires that you configure and build SLURM --with-pmix.
Versions earlier than 16.05: you must use either SLURM's PMI-1 or
PMI-2 support. SLURM builds PMI-1 by default, or you can manually
install PMI-2. You must then build Open MPI using --with-pmi pointing
to the SLURM PMI library location.
Please configure as appropriate and try again.
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[g225:3682368] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
I'm using an interactive srun job -c4 -n4 as options.
I have not tried using srun
directly on this image. Can you instead set up a batch script that you run with sbatch
or run the command interactively on a development node? The message seems to say that you need to compile MPI in a specific way to use srun
and our docker image was not set up that way.
@RobbieTheK Can I assume that using a batch script solved the problem?
No I ended up building and compiling candi and deal II. I'd be happy to try again with sbatch but I mentioned I did try with srun to no avail.
What I meant to ask is whether you found a way to make it work for you?
Yes, by installing deal II but it would be nice for future users to have a container option. Happy to test suggestions.
On Fri, Jul 14, 2023, 1:37 PM Wolfgang Bangerth @.***> wrote:
What I meant to ask is whether you found a way to make it work for you?
— Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geodynamics_aspect_issues_5137-23issuecomment-2D1636175496&d=DwMCaQ&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=-oUq7yBhhUP84-IIbiMP5GiJzBtM3RcKGAypJ39NpuOJkSFKWsr8BSgzEE1UDK8Y&s=p01DP47BowKsDRubj3EZ1hks-3ZMngRCRzL8w0rsBlg&e=, or unsubscribe https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AFBY5CZPWNUWHGXJ5GZL7U3XQF7WNANCNFSM6AAAAAAXBWGEGU&d=DwMCaQ&c=aqMfXOEvEJQh2iQMCb7Wy8l0sPnURkcqADc2guUW8IM&r=X0jL9y0sL4r4iU_qVtR3lLNo4tOL1ry_m7-psV3GejY&m=-oUq7yBhhUP84-IIbiMP5GiJzBtM3RcKGAypJ39NpuOJkSFKWsr8BSgzEE1UDK8Y&s=zrETeP0-UTXBBpPML191CrfZcEuIQ79iCPGf9iW8ST0&e= . You are receiving this because you were mentioned.Message ID: @.***>
I don't know that I have anything to offer. I don't know much about singularity (or containers in general) and I don't work on the TACC machines. I'm also not sure we have the resources as a project to really figure this out.
@gassmoeller @tjhei Do you have anything to offer? Or should we just say "We'd love to provide this, but we can't" and close the issue?
Using
geodynamics/aspect:latest-tacc
, with Singularity version 3.7.1 on RHEL 8 with OpenMPI 4.1.5a1I also took at shot at using the Docker image and pulling it into Singularity
That's likely from not having the same version of OpenMPI, am I correct?