ReproZip automatic shell/consolidated database

[remram44]

(high-level idea, needs brainstorming)

Similarly to noWorkflow or Git, the .reprozip-trace folder in the project directory would contain past versions of the experiment (inputs, outputs, dependencies, binaries (sources?)), in a queryable format, allowing the user to find old results, use new inputs on old software versions, query how an output file was created, run new code in a previous environment, etc.

To do that:

• [X] Come up with a better storage format, that would accept multiple versions of the same files (Git? file_archive? database+flat store)
• [X] Change C tracer to store files during execution
• [ ] New interface to automatically trace runs with their metadata in that database. RPZ packages would be created by extracting a subset of these runs/files from the database. Deleting runs would use some kind of garbage collection (like Git)
• [ ] Query metadata
• Needs new unpacker to easily be able to re-run past environments (setting up a separate Docker container is heavy, and will hog unnecessary disk space)

[remram44]

I started work on this in a new branch. Went with flat file storage + SQLite3 database, where all the metadata is stored (basically a table: run id, path, UNIX perms, content hash).

Looks very doable, but line between reprozip & reprounzip becomes fuzzier.