GiovanniBussi
commented
1 year ago Although I managed to run it for one replica, it seems like the plugin doesn't support multi replicas right now. I modified its source code and passed an MPI communicator from Python to C++ through Swig, then used
plumed_cmd(plumedmain, "setMPIComm", &comm);to set the communicator. Now, Plumed output shows that it is running on multiple nodes (whatever number I choose after the np flag when I runmpirun -np 4 python simulate.py), but still the number of replicas shown in the output is 1. I looked at the Gromacs patch and figured that I need to add another communicator, there should be inter and intra comms, is that right?
Yes, correct.
I passed another one and wrote the following piece of code to make use of the comms:
int done_already; int intra_comm_rank; MPI_Initialized(&done_already); MPI_Comm intra_comm = force.getIntracom(); MPI_Comm inter_comm = force.getIntercom(); MPI_Comm_rank(intra_comm, &intra_comm_rank); if (!done_already) { MPI_Init(NULL, NULL); } if (intra_comm_rank == 0) plumed_cmd(plumedmain, "GREX setMPIIntercomm", &inter_comm); plumed_cmd(plumedmain, "GREX setMPIIntracomm", &intra_comm); plumed_cmd(plumedmain, "GREX init");After the changes, the Python script hangs when I run it with MPI and doesn't show any output. However, if include the GREX commands inside the
if (!done_already), it shows the following error message:Traceback (most recent call last): File "/mnt/f/CALVADOS/single_chain/ACTR/mpi/simulate.py", line 149, in <module> simulation.minimizeEnergy() File "/home/hussein/miniconda3/envs/try4/lib/python3.9/site-packages/openmm/app/simulation.py", line 137, in minimizeEnergy mm.LocalEnergyMinimizer.minimize(self.context, tolerance, maxIterations) File "/home/hussein/miniconda3/envs/try4/lib/python3.9/site-packages/openmm/openmm.py", line 17208, in minimize return _openmm.LocalEnergyMinimizer_minimize(context, tolerance, maxIterations) openmm.OpenMMException: +++ PLUMED error +++ message follows +++ This command expects a floating point type. Received a void insteadShould the Intercommunicator be passed from the Python script just like the Intracommunicator? Is there something I am missing?
My guess is that MPI_Init is called outside, do that done_already is true, and you are just skipping those calls. I cannot say where the exception comes from (I realise not that this would be a good case for adding more context with a nested exception, but this requires modifying plumed). You can try to set export PLUMED_STACK_TRACE=1 and see the stack trace.
Is the error disappearing if you are not using MPI?
Regarding the communicators you are passing, the code seems correct (provided force.getIntracom() and force.getIntercom()` do the right thing).
In the case with the hanging run, can you find out where it's hanging (old-fashion debugging, putting std::cerr<<"A"<<std::endl; after every line in the C++ code)?
husseinmur
Hello,
I am trying to run a Metadynamics Metainference simulation with multi replicas using OpenMM and Plumed. I am using the OpenMM-Plumed plugin to add the Plumed script to the forcefield.
Although I managed to run it for one replica, it seems like the plugin doesn't support multi replicas right now. I modified its source code and passed an MPI communicator from Python to C++ through Swig, then used
plumed_cmd(plumedmain, "setMPIComm", &comm);to set the communicator. Now, Plumed output shows that it is running on multiple nodes (whatever number I choose after the np flag when I runmpirun -np 4 python simulate.py), but still the number of replicas shown in the output is 1. I looked at the Gromacs patch and figured that I need to add another communicator, there should be inter and intra comms, is that right?I passed another one and wrote the following piece of code to make use of the comms:
After the changes, the Python script hangs when I run it with MPI and doesn't show any output. However, if include the GREX commands inside the
if (!done_already), it shows the following error message:Should the Intercommunicator be passed from the Python script just like the Intracommunicator? Is there something I am missing?
Here are the relevant code snippet for reference:
The Swig file:
The Python script (snippet that involves MPI):
The Openmm-Plumed kernel: