Handling storage and logging for first-gen application

[jchodera]

We need to figure out how we want to implement a storage and logging system for the initial command-line relative free energy application.

[pgrinaway]

Initially, we want to store the following:

• For equilibrium trajectories of A and B states:
  • Trajectories. This is most easily done in a format mdtraj can write (e.g. HDF5, which can store full topologies, box volumes, velocities, energies, etc.)
  • Checkpoints so we can resume. If we store full explicit-solvent snapshots and box volumes in the trajectories, we can just resume from this.
• For nonequilibrium switching trials:
  • Nonequilibrium switching trajectories. For debugging purposes, it will be extremely valuable to visualize these trajectories. For actual production, we can just store the final snapshots, which will be still useful to analyze for determining how accurately nonequilibrium switching from A to B is modeling the bound ensemble of B.
  • Nonequilibrium work histories. For debugging purposes, it will be extremely valuable to store the work histories to help us optimize nonequilibrium thermodynamic protocols. For production, only the final work values are needed.

Initially, we might simply produce the following output files:

• equilibrium-{initial,final}.h5 - equilibrium trajectories of initial and final states with all atomic positions, velocities, box vectors, energies, etc.; can be used to resume simulation, with new iterations appended as frames
• nonequilibrium-{initial-to-final,final-to-initial}-00001.xtc - solute-only trajectories for nonequilubrium switching; here, 00001 is the iteration number, and would be incremented for each iteration
• nonequilibrium-{initial-to-final,final-to-initial}-00001.npy - work values for nonequilibrium switching

We also want to output some initial simulation setup data that tells us what calculation we ran:

• initially we can output the NonequilibriumFEPSetup to disk as a pickle

We also need some way to log/monitor run status:

• Logging status from each tasklet
  • Celery output
  • Log to redis
• Logging status from master

[jchodera]

@pgrinaway : I've updated the proposal above with a concrete suggestion as to how to store data. Would love your feedback/revisions, especially regarding how to store nonequilibrium data.

If the tasklets return, say, mdtraj.Trajectory objects and numpy arrays, a chained tasklet responsible for writing the data to disk can later be swapped out to deposit the data in a database or something.

[pgrinaway]

Being addressed in #396

[pgrinaway]

Addressed in #396