Speed/performance OpenMM with CDK2 example

[Oulfin]

Hi TIES developers,

I am trying to run RBFE with one transformation for CDK2 protein (ca 350 residues). When I am running the simulation I see that for the lig-complex it runs at 60 ns/day in a single GPU, but when I run a normal MD production with OpenMM it simulates at 400 ns/day. Is it normal such a decrease in performance? If I run 13 windows then the time remaining is 2 days with default options in only 1 replica.

Maybe I need to change something in the TIES.cfg file or in the command line ties_md --config_file=ties-1H1Q-1H1R/com/TIES.cfg --exp_name=complex --run_type=run --devices=0 --rep_id=0 to speed the simulation.

Thank you very much!

[adw62]

Hi,

This does not sound normal but I'm not sure the cause. Could you provide your ties-1H1Q-1H1R folder as a zip and I'll give a check and see if I can reproduce the problem?

Cheers, Alex

[Oulfin]

Hi,

Here are the files I have created. I just wanted 1 replica and 3 windows to test how fast is the performance, I am running in 1 gpu.

ties-1H1Q-1H1R.zip

Thanks!

Adrian

[adw62]

hmmm I get the same maybe 40ns/day on my 2070. However thinking about it again 60ns/day might be the right ball park for a protein ligand complex and 400ns/day might be more typical just for the solvated ligand. If you don't mind would it please be possible to post the normal MD scripts as well just for a sanity check?

[Oulfin]

Hi,

Yes you are right. Actually maybe it goes faster because I use different options like:

Hydrogen mass partition, time step 4 femtoseconds, LangevinMiddleIntegrator, nonbondedCutoff=0.9*nanometer and in ambertools I solvated at 10 Å not 14 Å. I think all of this increase the speed of the performance, have you tested or is possible to change this options?

Another question, do you also get "Time Remaining" of 2 days? This happened when I used the 13 windows. Is there any post-processing data that makes the whole process slow down? like when doing this "Propagating iteration 224/500 in window 3/3 for replica 0" ?

Thanks!

[adw62]

Hi,

The settings could be changed here for the MD: https://github.com/UCL-CCS/TIES_MD/blob/main/TIES_MD/alch.py#L116-L117

and I think here for the system setup: https://github.com/UCL-CCS/TIES/blob/master/ties/scripts/ambertools/leap_complex.in#L16

To change things you can clone the repo make the changes then pip install the new code.

With regards to long simulation times I think this is just a result of all the sampling that is needed (around 390ns typically). Ideally each calculation would use 65 GPUs (one per window and replica) and then you can get results in a few hours. This is not always practical though, the alternative is to trim down the calculations as much as possible and we wrote an paper on how this could be done here: https://chemrxiv.org/engage/api-gateway/chemrxiv/assets/orp/resource/item/62eb9531cdd9187dc50b006f/original/which-corners-to-cut-guidelines-on-choosing-optimal-settings-to-maximise-sampling-with-limited-computational-resources.pdf

Generally there is some 'post processing' being done that slows the calculation compared to normal MD, and this is stopping to calculate the energies and gradients needed for alchemical methods here: https://github.com/UCL-CCS/TIES_MD/blob/main/TIES_MD/alch.py#L886-L902 However, this is not done every timestep and the overhead is generally small for larger systems like these protein ligand complexes.

[Oulfin]

I see, thank you very much for the help. All questions are answered. I close this issue.

[adw62]

Just to close this out, I've added an option --fast=True which can be added to the ties_md run line. This will use the fast settings you have suggested here with the recommended OpenMM settings and gives a 2.5x speed increase.

The largest contributors to this were changing the Ewald tolerance from 0.00001 to 0.0005 and increasing the time step from 2fs to 4fs. This deviates from our validated protocol but the speed increase is probably worth it.