Device index not passed for openmm engine

[sroet]

When running the openmm engine with {'OpenCLDeviceIndex' : '1'} set, it always runs on device 0 instead of the given DeviceIndex. This is the full engine generating script.

from simtk.openmm import app
from simtk import unit
import openmmtools as omt
import simtk.openmm as mm
import openpathsampling as paths
import openpathsampling.engines.openmm as peng_omm

template = peng_omm.snapshot_from_pdb('input.pdb')
topology = template.topology
gro = app.GromacsGroFile('input.gro')
box_vectors = mm.Vec3(*gro.getPeriodicBoxVectors())

forcefield = app.GromacsTopFile('topol.top', periodicBoxVectors=box_vectors)

system = forcefield.createSystem(
        nonbondedMethod=app.PME,
        nonbondedCutoff=1.1*unit.nanometers,
        constraints=app.HBonds,
        rigidWater=True,
        ewaldErrorTolerance=0.0005
)

system.addForce(
        mm.MonteCarloBarostat(
                    1*unit.atmospheres,
                    310*unit.kelvin,
                    25
                )
)

integrator = omt.integrators.VVVRIntegrator(
        310*unit.kelvin,
        5.0/unit.picoseconds,
        2.0*unit.femtoseconds
)

integrator.setConstraintTolerance(0.00001)

openmm_properties = {'OpenCLDeviceIndex': '1'}

options = {'n_frames_max': 4000,
           'n_steps_per_frame':2500,
           'platform':'OpenCL'}

engine = peng_omm.Engine(
        template.topology,
        system,
        integrator,
        openmm_properties = openmm_properties,
        options=options
)

This option does work when using pure openmm, so I think there's a strange issue passing the openmm_properties

[jhprinz]

Okay, I see the problem and checked that the options are passed to the engine. The problem is in the way you initialize the engine. We changed this a little so let me explain with the example that should work. Keep all you had except

openmm_properties = {'OpenCLDeviceIndex': '1'}

options = {'n_frames_max': 4000,
           'n_steps_per_frame':2500}

engine = peng_omm.Engine(
        template.topology,
        system,
        integrator,
        openmm_properties = openmm_properties,
        options=options
)

engine.initialize(platform='OpenCL')

if you do not specify the platform when initializing it will use the fastest available and this could be CUDA.

At least check that my example works.

Now the bigger problem I see here or maybe you (@sroet and @dwhswenson) have an idea:

The issue

We agreed that we did not want to fix completely the platform, which allows to restart the same saved engine on another platform. Hence the initialisation with the platform needed. The current problem is that we store platform specific options which render the approach unfeasible. Fortunately the CUDA platform does not care about non-CUDA options and will ignore them. The other platforms will complain (this is to be changed in the next OpenMM release).

I think the current way does not make (complete) sense we either

1. stick to the old way and really create an engine with all the options and recreate the engine anytime we want to run on another machine with another platform, etc... This is the easiest way, but it will also mean to recreate movers that use the engine.

2. We separate the engine description and the actual instance of the engine. I had the idea before and it would mean the the Engine is only a kind of factory which the instance independent properties and running the initialization can set properties of the engine for a particular instance. If you use the engine it will create an EngineInstance object that contains the additional information like machine, time of creation, additional properties, etc... and this is what is stored with the snapshots instead of Engine itself. This could in particular be useful when using Gromacs or a general ExternalEngine to store information on the specific run.

[sroet]

@jhprinz

At least check that my example works.

This works! Thanks! I'm not entirely sure why it wasn't before though. I understand that the "fastest" platform is always OpenCL device 0 (CUDA isn't an available platform for the OpenMM build that I'm currently using) , but the OpenCLDeviceIndex should be able to overwrite this, right?

[jhprinz]

That is strange actually. The only thing I can imagine is that Openmm will ignore platformProperties in case you do not specify a platform. This would make sense from the point that properties are not compatible between platforms.

[sroet]

It also behaves unexpected when I try to restart. Running the following code defaults to device 0, even when the original engine ran on device 1 (in a different script).

def next_steps(old_storage, current_storage, n_steps):
    tps_sampler = old_storage.pathsimulators[0]
    tps_sampler.storage = current_storage
    tps_sampler.restart_at_step(old_storage.steps[-1])
    tps_sampler.run(n_steps)

[jhprinz]

Its been a while. Any news here? I did not have much time to look into this. There is still the issue that when restarting after loading you need to initialise an engine using .initialize()

[sroet]

Current status of this is: if I load the engine and do analysis with it, without initializing it explicitly it runs on device 0. With explicit .initialize(platform='OpenCL') it seems to run on the original device index of 1. I didn't have the time yet to look how this behaves in the case of me trying to restart a tps simulation