should work with Gromacs 5.x

[orbeckst]

MDPOW should be using Gromacs 5.1.x, which is now pretty stable. GromacsWrapper recently moved to Gromacs 5.x Becksteinlab/GromacsWrapper#26 so this should work with the latest GW.

[ianmkenney]

I cannot seem to get MDPOW to play well with gromacs 5.1.1. The process breaks when energy minimizing. The following command is invoked: gmx mdrun_d ... which I cannot get to work. I have tried compiling a double precision version of gromacs. But cannot get the command to work. I cannot find documentation on the command either... The command seems to be coming from GromacsWrapper. How should I go about fixing this?

[orbeckst]

Can you invoke

gmx_d mdrun

(It actually depends on what suffix you chose when you compoiled Gromacs in double).

You'd need to define an appropriate entry for the double precision binaries.

However, MDPOW should just fall back to standard gmx mdrun.

Post the whole stacktrace to see where things go wrong.

[ianmkenney]

Here is the traceback

mdpow/tests/test_equilibration.py EEE

================================= ERRORS =================================
________ ERROR at setup of TestEquilibration.test_simulation_file ________

self = <test_equilibration.TestEquilibration object at 0x7f454d4ad550>

    def setup(self):
        self.basedir = os.getcwd()
        self.directory_name = tempfile.mkdtemp()
        print("basedir:  {}".format(self.basedir))
        print("Copying required files into:  {}".format(self.directory_name))
        files = ["benzene.itp","benzene.pdb"]
        for f in files:
            shutil.copy("mdpow/tests/benzene/{}".format(f),"{0}/{1}".format(self.directory_name,f))
            print("\tbenzene/{0} --> {1}/{0}".format(f,self.directory_name))
        os.chdir(self.directory_name)
        print("Changed directory:  {}".format(os.getcwd()))
        for k in self.sims.keys():
            W = self.sims[k](molecule="BNZ")
            W.topology(itp="benzene.itp")
            W.solvate(struct="benzene.pdb")
>           W.energy_minimize()

/home/ikenney/Library/python/MDPOW/mdpow/tests/test_equilibration.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/ikenney/Library/python/MDPOW/mdpow/equil.py:369: in energy_minimize
    params = gromacs.setup.energy_minimize(**kwargs)
/home/ikenney/Library/python/GromacsWrapper/gromacs/setup.py:684: in energy_minimize
    gromacs.mdrun_d(**mdrun_args)
/home/ikenney/Library/python/GromacsWrapper/gromacs/core.py:382: in __call__
    return self.run(*args,**kwargs)
/home/ikenney/Library/python/GromacsWrapper/gromacs/core.py:160: in run
    results, p = self._run_command(*_args, **_kwargs)
/home/ikenney/Library/python/GromacsWrapper/gromacs/core.py:587: in _run_command
    self.check_failure(result, command_string=p.command_string)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <gromacs.tools.Mdrun_d object at 0x7f454d9a2310>
result = (1, '', '                      :-) GROMACS - gmx, VERSION 5.1.1 (-:\n\n                            GROMACS is written ...CS\nwebsite at http://www.gromacs.org/Documentation/Errors\n-------------------------------------------------------\n')
msg = 'Gromacs tool failed\nCommand invocation: gmx mdrun_d -c em.pdb -deffnm em -stepout 10 -v'
command_string = 'gmx mdrun_d -c em.pdb -deffnm em -stepout 10 -v'

    def check_failure(self, result, msg='Gromacs tool failed', command_string=None):
        rc, out, err = result
        if not command_string is None:
            msg += '\nCommand invocation: ' + str(command_string)
        had_success = (rc == 0)
        if not had_success:
            gmxoutput = "\n".join([x for x in [out, err] if not x is None])
            m = re.search(self.gmxfatal_pattern, gmxoutput, re.VERBOSE | re.DOTALL)
            if m:
                formatted_message = ['GMX_FATAL  '+line for line in m.group('message').split('\n')]
                msg = "\n".join(\
                    [msg, "Gromacs command %(program_name)r fatal error message:" % m.groupdict()] +
                    formatted_message)
            if self.failuremode == 'raise':
>               raise GromacsError(rc, msg)
E               GromacsError: [Errno 1] Gromacs tool failed
E               Command invocation: gmx mdrun_d -c em.pdb -deffnm em -stepout 10 -v

[orbeckst]

Can you get gromacs.setup.energy_minimize() to run manually with Gromacs 5.1.x?

If not file an issue for GW.

Oliver Beckstein email: orbeckst@gmail.com

Am Jun 3, 2016 um 21:14 schrieb Ian Kenney notifications@github.com:

Here is the traceback

mdpow/tests/test_equilibration.py EEE

[orbeckst]

Becksteinlab/GromacsWrapper#65 with Becksteinlab/GromacsWrapper#64 and Becksteinlab/GromacsWrapper#63 first need to be fixed before we can make progress here.

[orbeckst]

In principle, MDPOW should "just work"™ depending on what GromacsWrapper provides... so change of issue title.

[orbeckst]

I fixed the issue with energy_minimize() Becksteinlab/GromacsWrapper#63. Can you try with the latest GW develop?

[iorga]

@orbeckst, @ianmkenney :

I am trying to start a simulation with the tip5p water. In my PATH there is only Gromacs 4.6 and with the latest develop version I've got an error, apparently MDPOW is using the Gromacs 5.x syntax now. All other sims were done with Gromacs 4.6.5, so it will be good to use the same version till the end. Can you implement a way allowing to choose between Gromacs 4.x and 5.x ?

gromacs.core: ERROR    gmx editconf -f /gem/iorga/projects/sampl5_sims/V01-n-pentane/resources/V01-n-pentane.pdb -o boxed.gro -bt dodecahedron
ERROR:gromacs.core:gmx editconf -f /gem/iorga/projects/sampl5_sims/V01-n-pentane/resources/V01-n-pentane.pdb -o boxed.gro -bt dodecahedron
gromacs.core: CRITICAL Failed to find Gromacs command 'editconf', maybe its not on PATH or GMXRC must be sourced?

[orbeckst]

Hm, ok, will look into it. Can you check your .gromacswrapper.cfg and make sure that it contains Gromacs 4.6.x commands?

[iorga]

I don't have a .gromacswrapper.cfg, only the .gromacswrapper folder with the qscripts.

[orbeckst]

Then you will need one: I included a template 4.6.x file gromacswrapper_465.cfg in the repository: You can "install" it with

curl https://raw.githubusercontent.com/Becksteinlab/GromacsWrapper/develop/gromacs/templates/gromacswrapper_465.cfg -o ~/.gromacswrapper.cfg

This is required because in the absence of a cfg file, GromacsWrapper 0.5.x is defaulting to Gromacs 5.

[iorga]

Thanks @orbeckst, this effectively restores the 4.6.x syntax. But then, I get another error, which is the same for the 5 types of water (see below). The last line of the boxed.gro file is

   0.00000   0.00000   0.00000

whereas before it used to be

   2.67766   2.67766   1.89339   0.00000   0.00000   0.00000   0.00000   1.33883   1.33883

I could find that it comes from the missing parameter -d 1.0 at the end of the editconf command. Could you please restore it?

I also saw that the default value for this parameter is 0.9 in GromacsWrapper. I think that it is important to keep the same values for this and all other parameters (except for the change in the solvent parameters) to be able to compare realistically only the influence of the water type and not something else.

gromacs.utilities: INFO     Working in '/gem/iorga/projects/sampl5_sims/V01-n-pentane/V01-n-pentane/Equilibrium/water/solvation' (newly created)...
INFO:gromacs.utilities:Working in '/gem/iorga/projects/sampl5_sims/V01-n-pentane/V01-n-pentane/Equilibrium/water/solvation' (newly created)...
gromacs.setup: INFO     [/gem/iorga/projects/sampl5_sims/V01-n-pentane/V01-n-pentane/Equilibrium/water/solvation] Solvating with water 'tip4p.gro'...
INFO:gromacs.setup:[/gem/iorga/projects/sampl5_sims/V01-n-pentane/V01-n-pentane/Equilibrium/water/solvation] Solvating with water 'tip4p.gro'...
DEBUG:gromacs.core:editconf -f /gem/iorga/projects/sampl5_sims/V01-n-pentane/resources/V01-n-pentane.pdb -o boxed.gro -bt dodecahedron
DEBUG:gromacs.core:genbox -cs tip4p.gro -p /gem/iorga/projects/sampl5_sims/V01-n-pentane/V01-n-pentane/Equilibrium/water/top/system.top -cp boxed.gro -o solvated.gro
Traceback (most recent call last):
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/bin/mdpow-equilibrium", line 71, in <module>
    S = equilibrium_simulation(cfg, opts.solvent, dirname=opts.dirname)
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/mdpow/run.py", line 230, in equilibrium_simulation
    S.solvate(struct=cfg.getpath("setup", "structure"), maxwarn=maxwarn)
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/mdpow/equil.py", line 331, in solvate
    params = gromacs.setup.solvate(**kwargs)
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/gromacs/setup.py", line 503, in solvate
    gromacs.genbox(p=topology, cp='boxed.gro', cs=water, o='solvated.gro')
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/gromacs/core.py", line 380, in __call__
    return self.run(*args,**kwargs)
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/gromacs/core.py", line 160, in run
    results, p = self._run_command(*_args, **_kwargs)
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/gromacs/core.py", line 585, in _run_command
    self.check_failure(result, command_string=p.command_string)
  File "/sge/packs_centos/opt/python-2.7.10-CentOS6/lib/python2.7/site-packages/gromacs/core.py", line 540, in check_failure
    raise GromacsError(rc, msg)
gromacs.exceptions.GromacsError: [Errno 255] Gromacs tool failed
Command invocation: genbox -cs tip4p.gro -p /gem/iorga/projects/sampl5_sims/V01-n-pentane/V01-n-pentane/Equilibrium/water/top/system.top -cp boxed.gro -o solvated.gro
Gromacs command 'genbox' fatal error message:
GMX_FATAL  Source code file: /tmp/gromacs-4.6/src/tools/gmx_genbox.c, line: 906
GMX_FATAL  
GMX_FATAL  Fatal error:
GMX_FATAL  Undefined solute box.
GMX_FATAL  Create one with editconf or give explicit -box command line option
GMX_FATAL  For more information and tips for troubleshooting, please check the GROMACS
GMX_FATAL  website at http://www.gromacs.org/Documentation/Errors

[orbeckst]

Should be fixed in b0e3b1895b4c889e3b3d536fc458a8ab88a76e08 – sorry. Not sure why I didn't catch it before when I ran tests for the new water models. I just tried benzene with TIP4P and TIP5P and it seems to work.

I think @ianmkenney and I thought about increasing the water distance to 1.2 nm to avoid cutoff problems in very small boxes but I set it back to the value of 1.0 nm that we used for SAMPL5.

[iorga]

Thanks @orbeckst, it works now.

[orbeckst]

The tests pass with Gromacs 4 and 5 but coverage is still low so I leave it open until we are confident that we get sensible results with either one.

[ianmkenney]

@orbeckst I am running into problems getting 5.1.2 to pass tests while 4.6.7 seems fine. I suspect that it's a gromacs wrapper issue rather than an MDPOW issue. https://travis-ci.org/Becksteinlab/MDPOW/builds/155678112.

Do you see any reason why I should exclude 5.1.2 from the tests while developing them? Since MDPOW should be GMX version agnostic, I see no theoretical downsides to doing this. I would of course start testing it again once the main tests are written.

[orbeckst]

Only one test failed but I don't see why. Can you run the test manually?

Because everything else works, it seems that in principle Gromacs 4 and Gromacs 5 should work. If everything with 5 failed I'd say put 5 aside for the time being and make it work later. However, it looks like something small so you'd gain much more by understanding now what the problem with 5 is and then continue writing tests that pass on 4 and 5.

So, I'd say: spend some time figuring out why this one test failed with 5.

On 27 Aug, 2016, at 22:35, Ian Kenney notifications@github.com wrote:

@orbeckst I am running into problems getting 5.1.2 to pass tests while 4.6.7 seems fine. I suspect that it's a gromacs wrapper issue rather than an MDPOW issue. https://travis-ci.org/Becksteinlab/MDPOW/builds/155678112.

Do you see any reason why I should exclude 5.1.2 from the tests while developing them? Since MDPOW should be GMX version agnostic, I see no theoretical downsides to doing this. I would of course start testing it again once the main tests are written.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Oliver Beckstein * orbeckst@gmx.net skype: orbeckst * orbeckst@gmail.com

[ianmkenney]

I was having trouble finding the error using the tests directly. I decided to run the equilibrium script directly. This is what I got. I am using the gromacs file packaged with gromacswrapper. The output from the tests was the same except for the traceback.

INFO:mdpow.run:Run directory: /tmp/tmpHd8deB/benzene/Equilibrium/water/MD_relaxed
Traceback (most recent call last):
  File "/nfs/homes/ikenney/.local/bin/mdpow-equilibrium", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/nfs/homes/ikenney/Library/python/MDPOW/scripts/mdpow-equilibrium", line 71, in <module>
    S = equilibrium_simulation(cfg, opts.solvent, dirname=opts.dirname)
  File "/nfs/homes/ikenney/Library/python/MDPOW/mdpow/run.py", line 245, in equilibrium_simulation
    success = wrapper(runMD_or_exit, S, "MD_relaxed", params, cfg)  # note: MD_relaxed!
  File "/nfs/homes/ikenney/Library/python/MDPOW/mdpow/restart.py", line 217, in dummy_protocol
    success = func(*args[1:], **kwargs)
  File "/nfs/homes/ikenney/Library/python/MDPOW/mdpow/run.py", line 147, in runMD_or_exit
    cpi=True, append=True)
  File "/nfs/homes/ikenney/.local/lib/python2.7/site-packages/gromacs/run.py", line 139, in __init__
    self.driver, self.name = find_gromacs_command(self.mdrun)
  File "/nfs/homes/ikenney/.local/lib/python2.7/site-packages/gromacs/run.py", line 70, in find_gromacs_command
    raise OSError(errno.ENOENT, "No Gromacs executable found in", ", ".join(commands))
OSError: [Errno 2] No Gromacs executable found in: 'm, d, r, u, n'

[orbeckst]

That's a GW bug. Somehow it treats a single string as a list. Probably should be wrapped in util.asiterable().

Oliver Beckstein email: orbeckst@gmail.com

Am Aug 28, 2016 um 15:02 schrieb Ian Kenney notifications@github.com:

I was having trouble finding the error using the tests directly. I decided to run the equilibrium script directly. This is what I got. I am using the gromacs file packaged with gromacswrapper. The output from the tests was the same except for the traceback.

INFO:mdpow.run:Run directory: /tmp/tmpHd8deB/benzene/Equilibrium/water/MD_relaxed Traceback (most recent call last): File "/nfs/homes/ikenney/.local/bin/mdpow-equilibrium", line 6, in exec(compile(open(file).read(), file, 'exec')) File "/nfs/homes/ikenney/Library/python/MDPOW/scripts/mdpow-equilibrium", line 71, in S = equilibrium_simulation(cfg, opts.solvent, dirname=opts.dirname) File "/nfs/homes/ikenney/Library/python/MDPOW/mdpow/run.py", line 245, in equilibrium_simulation success = wrapper(runMD_or_exit, S, "MD_relaxed", params, cfg) # note: MD_relaxed! File "/nfs/homes/ikenney/Library/python/MDPOW/mdpow/restart.py", line 217, in dummy_protocol success = func(_args[1:], *_kwargs) File "/nfs/homes/ikenney/Library/python/MDPOW/mdpow/run.py", line 147, in runMD_or_exit cpi=True, append=True) File "/nfs/homes/ikenney/.local/lib/python2.7/site-packages/gromacs/run.py", line 139, in init self.driver, self.name = find_gromacs_command(self.mdrun) File "/nfs/homes/ikenney/.local/lib/python2.7/site-packages/gromacs/run.py", line 70, in find_gromacs_command raise OSError(errno.ENOENT, "No Gromacs executable found in", ", ".join(commands)) OSError: [Errno 2] No Gromacs executable found in: 'm, d, r, u, n' — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[orbeckst]

@ianmkenney please raise an issue in the GW tracker and reference https://github.com/Becksteinlab/MDPOW/issues/36#issuecomment-243002309.

[orbeckst]

@ianmkenney can you check with latest GW develop? I fixed Becksteinlab/GromacsWrapper#85.

[orbeckst]

Actually, try with Gromacs 0.6.0, just released.

(The tests should not require any .gromacswrapper.cfg anymore, just sourcing GMXRC will be sufficient to load either Gromacs 4 or Gromacs 5.)

[orbeckst]

See PR #65 for test results. I'll do the discussion on the PR.

[orbeckst]

The tests pass locally with 5.1.2 but on Travis the mdpow/tests/test_equilibration_script.py appears to hang when it does not produce any output after 10 min it gets killed by Travis. The same job finishes in 2:30min under Gromacs 4.6.6 so I suppose something weird happens with gmx mdrun.

Possibly run the tests with more verbose py.test/travis output?

Can you please have a look @ianmkenney ?

[ianmkenney]

@orbeckst I ran the tests on travis using py.test -s mdpow mdpow/tests. This disables capturing from pytest. As previously stated, the tests hang on mdpow/tests/test_equilibration_script.py. What's interesting is that it hangs on the second gmx mdrun invocation. In response to this, I decided to run the tests using our new gromacs-2016 conda package; this yields the exact same behavior.

From here, I decided to build a docker image that replicates TravisCI testing environments (I will be posting that soon on github; in the meantime ~ikenney/Projects/docker/local-travis/src) to run tests locally and possibly figure out the problem. Both gromacs-2016 and gromacs-5.1.2 passed.

I am at a loss at this point. The thing I can think of is the fact that during the tests, the speed of energy minimization on 2016 and 5.1.2 is slow enough that it doesn't finish by the time 4.6.7 finishes all of its tests. Any ideas?

[orbeckst]

On 4 Jan, 2017, at 22:09, Ian Kenney notifications@github.com wrote:

I am a bit of a loss at this point. The thing I can think of is the fact that during the tests, the speed of energy minimization on 2016 and 5.1.2 is slow enough that it doesn't finish by the time 4.6.7 finishes all of its tests. Any ideas?

If your hypothesis is that the time taken to do the energy minimization is the problem then change the test so that the energy minimization finishes in a few seconds. Decrease the maximum number of steps in the MDP file.

Initially, you don't even need to adapt the tests – just seeing it not hang and then gloriously fail would be a success.

[orbeckst]

@ianmkenney did fixing the number of threads (-nt 2) improve things – given that Gromacs 5.x/2016 apparently used 32 threads on travis and Gromacs 4.6.x still used 12 (even though the Travis Docker container should only provide 2 virtual cores)?

[orbeckst]

Yay! Thanks @ianmkenney ... finally done. And as a bonus, it also works with Gromacs 2016 and we figured out how to run the tests faster on travis.