search

open-research / sumatra

http://neuralensemble.org/sumatra/
BSD 2-Clause "Simplified" License
127 stars 48 forks source link

Replication of records - Using Sumatra to publish computational projects #379

Open felix11h opened 6 years ago

felix11h commented 6 years ago

I'm interested in using Sumatra to publish computational projects (example here). The smt repeat function is very helpful in repeating computations under original conditions and testing if the repeated computation matches the original record. For this, a previous state of the project (e.g. git version, potentially other parameters?) is loaded.

For a published computational project, a reader might interested in replicating (single) results in a straightforward manner. For this smt repeat could be great. However, the loading of a previous version of the code "behind the scenes" can be very misleading. Old versions of the code might look very different from what the reader is seeing in the published repository.

I'm suggesting either an option for smt repeat or a new, related command smt replicate. The command would function similarly to smt repeat but instead of checking out a previous version of the project, the command repeats the record with the current version. This would allow a test whether the code, as currently shown, is able to replicate the results obtained originally.

To allow for complete or partial replication of the project, the command would take either a record, a tag or the whole project and replicate all records in the group in chronological order. The tag command specifically could help replicate the just the simulation, data-analysis or plotting stages in the project, as for example described in the Popper convention (cc @ivotron).

I'm interested in feedback on this. Implementation of such functionality seems, at first glance, manageable. Am I overlooking expected complications? Would such functionality be seen as helpful and fitting within the scope of Sumatra?

apdavison commented 6 years ago

I think this is a great idea; it ought to be fairly straightforward to implement, at least for re-running individual stages; re-running entire workflows is a more complex issue and probably needs its own smt command (cf #107)

I would suggest smt repeat --update as the command. The usage of "replicate" vs "reproduce" is still being argued, and I would prefer to avoid getting into naming arguments.