Issues with domain decomposition?

[yummy-hat]

Hello,

Thanks again for sharing this script. It worked well for my system and I've been able to generate some trajectories.

I'm running into a domain decomposition issue. In the OLIVES script output, I noticed the following:

This can in lead to problems with domain decomposition if molecules break apart (usually only happens for quartenary structure which could a be desired property). In these cases it is best to use all openMP threads instead of domain decomposition, see: https://manual.gromacs.org/documentation/current/user-guide/mdrun-performance.html

Could you explain this a bit or share your command-line options? The shell scripts in the examples folder have minimal options for mdrun. And generally, I have been struggling a bit getting good efficiency for these Martini3 systems with Go model on top.

[KasperBuskPedersen]

Hi @yummy-hat

I would be interested in knowing what type of system you are experiencing these errors for and see your log file. Note that this usually means that either you have a protein complex that dissociates or a protein that unfolds, so just make sure you intent this to happen. If you don't want to to share it here you can email me at kasperbusk@chem.au.dk

If you get domain decomposition (DD) errors, there are a few things you can try.

1. Use the mdrun flag "-noddcheck" together with "-rdd 2.0". "-noddcheck" turns off the error for pairs that become too long, abruptly ending your run (beyond -rdd, which is usually automatically found by mdrun). The 2 nm in -rdd is where a LJ potential with minimum distance of 0.55 nm (olives cutoff) goes to 0.

2. Turn off DD by setting number of MPI processes to 1. On a single node this can be done using the mdrun flag "-ntmpi 1". This can result in a performance loss depending on your hardware setup.

Let me know if one of these solutions work for you. Also make sure to read our discussion on the topic in the OLIVES paper.

I am afraid that I cannot help much with optimizing your mdrun performance in general, since this depends a lot on your hardware. Usually it involves trying different partitions between MPI and openMP processes. OLIVES generally runs faster than GoMARTINI and should also be competitive with elastic networks.

Best regards Kasper

[yummy-hat]

Hi Kasper,

Thank you, this is exactly what I needed. I was missing the "-noddcheck" flag. I wasn't able to test the other solution on our supercomputer because it's installed without thread-MPI (I can't set the number of MPI threads).

So, if I'm understanding correctly, the crash I encountered before was just because one of my OLIVES bonds got too large for the domain decomposition to handle?

Not sure if this is an appropriate place to ask, but do you have any sense for whether force could be applied to protein systems held together by OLIVES bonds?

[KasperBuskPedersen]

@yummy-hat

So, if I'm understanding correctly, the crash I encountered before was just because one of my OLIVES bonds got too large for the domain decomposition to handle?

Yes, it is because LJ 1-4 interactions in GROMACS are regarded as bonded interactions and there is an error check that bonded interactions don't get too large relative to a domain. Because separate domains are calculated independently and only communicated at an interval, too long bonded interactions could get missed if spanning multiple domains. Since the LJ potential goes to 0 at longer bond lengths, it is not a problem that the force contribution is not calculated for long LJ bonds. I suggested -rdd 2.0 to force a minimum length of domains, to be extra sure that any force contribution from OLIVES LJ potentials doesn't get missed.

Not sure if this is an appropriate place to ask, but do you have any sense for whether force could be applied to protein systems held together by OLIVES bonds?

Sure you can. I am working on an example of how to use the GROMACS pull code to pull ubiquitin on both N- and C-termini, unfolding it (https://manual.gromacs.org/current/reference-manual/special/pulling.html). Alternatively, you could use steered dynamics with PLUMED. OLIVES is ideally suited to tackle this problem in Martini 3, since the DSSP restraints required for GoMARTINI in Martini 3 effectively make it impossible to pull a beta-sheet apart (note that this was not the case in Martini 2 and Poma et al. studied unfolding in M2 in their original work https://pubs.acs.org/doi/full/10.1021/acs.jctc.6b00986). To get you started I have attached my mdp file for COM pulling using GROMACS. Note that you will have to create a .ndx file for the pull group names. pull.zip