Open benmwebb opened 7 years ago
For tracking provenance of most experimental information, some additional information needs to be stored in the RMF file, namely the set of restraints, which particles they act on, and which restraints were used in each sampling step.
Proposal: RMF already stores basic information about decomposed restraints. Make each set of decomposed restraints children of the 'real' restraint, which holds serialized information on the restraint itself (e.g. filename where the EM map was read from, cross correlation information, total score). The SampleProvenance
decorator then contains an RMF Alias node child for each restraint used in that sampling. This information is already stored in IMP (partly in the Model
, and partly in the ScoringFunction
.)
IMP currently takes as input files in a variety of formats, but doesn't care where those files originate. This becomes a problem when we come to publish a modeling study and deposit the files (e.g. at PDB-dev). It's a lot of work to backtrack and try to figure out where such files came from. It would be much simpler if IMP tracked this information from day one, reading it in some standardized way from the files themselves (or the Python script), storing it in the Model, and also storing it in RMF files.
Since this is prerequisite information for outputting mmCIF files, solving this issue would be a step towards addressing #968. Much of this information is currently stored outside of the Model, mostly in PMI 1 data structures, and so currently outputting mmCIF requires PMI 1.
Only input atomic models are explicitly considered here but similar considerations should apply to restraints (e.g. where an EM map comes from), sequences (e.g. uniprot identifier), etc. (More generally, any transformation of the model, such as sampling, filtering or clustering, should also be recorded.)
Input files
atom::read_pdb
.mrc
or.pgm
files) the metadata will need to be stored somewhere else - one solution would be an accompanying JSON file (e.g.foo.mrc
is described byfoo.mrc.json
) with domain-specific metadata.Storage in Model
atom::read_pdb
should add suitable decorators to the createdHierarchy
particles to identify their source.atom::StructureSource
(see #894) is one example, although more data should be stored here (e.g. path to the file, PDB ID, version, descriptive text).atom::create_simplified_along_backbone
and PMI's generation of initial models should copy or otherwise preserve this information in the newly-generated hierarchies, while adding additional information about the simplification applied (e.g. the resolution).core::ProvenanceHierarchy
decorator to track a separate hierarchy from theatom::Hierarchy
. The root of this tree is the current state of the object, while children are inputs or previous states (and so will be bothcore::ProvenanceHierarchy
and some other decorator such asStructureSource
). A particle can be decorated as bothcore::ProvenanceHierarchy
andatom::Hierarchy
. Example hierarchies include:System
(atom::Hierarchy
), alsocore::ProvenanceHierarchy
rootoutput.1/rmfs/0.rmf3
foo.pgm
foo.csv
output.2/rmfs/0.rmf3
Chain
(atom::Hierarchy
), alsocore::ProvenanceHierarchy
rootfoo.pdb
, chainA
, heavy atoms only1xyzA
with alignmentfoo.ali
core::ProvenanceHierarchy
should be static (i.e. the same for all frames in a trajectory) so that it doesn't need to be updated during a simulation, and can be stored efficiently in an RMF file. Care also needs to be taken to avoid unnecessary duplication (e.g. each ensemble contributing to a cluster likely has the exact same set of inputs).Storage in RMF