Design input-tracking db/git archive/retrieval system

[billyc]

In addition to logging runs, we also need to archive the inputs so we can immediately restore and rerun any run at any time.

We need to think this through; but my initial thought is that Git is actually well-suited to this and might be better than a database design.

How Git might work:

• Maybe just one "inputs repository" is created per model type. So currently we'd have a soundcast inputs repo and a 4k inputs repo. Each run is just stored linearly in the master branch. That would make it really easy to spot differences vs. a known-good or base run, for example.
• Base run inputs could be tagged with useful labels
• The model-run script would need one additional step: after copy-inputs, take a snapshot of the working folder and commit it to the git inputs repo. The commit author field would be the name of user who spawned the run; the comment could list the project or notes from the run log.
• Large files and non-text files such as parcel buffer data, exe's, are hard to diff and store, but git extensions such as git-fat and git-lfs probably help us here. Git-fat in particular is super easy to set up and just requires a common storage folder locally. We don't want our inputs outside the firewall anyway. (See https://github.com/jedbrown/git-fat ) This would need testing.
• Git has the big advantage of only storing a "blob" once for files that don't actually change; so while parcel-data objects are big, they're not copied over and over in the repository.
• From the runlog system, you could instantly restore a run with just a link to the git commit. One "git checkout [tag]" later and you have everything. (We'd have to think through disabling the re-copying of inputs in the model run script if they already exist, though)

  How would a database approach work?

• Would need to talk to Stefan, Andy, others about a db approach.

[Ennazus]

Thanks for writing this up. Tracking inputs and outputs is one of the most major challenges in travel modeling, usually because we can't find enough storage space to let things sit for a while. I hadn't heard of gitfat. It does seem like the file size and culture shift would be the biggest hurdles.

[billyc]

i'm pretty sure the method above is going to work. let's try it. some details, so i don't forget later:

• dashboard server will create one repository per tool (soundcast/4k/etc), stored on the network somewhere
• soundcast/4k tool will add an extra step "archive-this-run.py/.bat" at the appropriate point after copying all inputs.
• archive-this-run will:
  • use rsync to copy all changed files to the repository
  • git add everything
  • git commit using the current user's name and the project name/series in the comment field
• I think that's it!

we'll have to test to make sure the removed files/folders work properly and not just additions. (i don't know rsync very well)

Rsync won't bother copying files that haven't changed, and git won't store duplicates of files that haven't changed, so overall this shouldn't add much size or time to the process.

To restore a run, you can fetch the repository itself and check out a particular commit, or we can add a feature to download a .zip file of the entire model inputs folder to the dashboard. oooh yeah!

[billyc]

reading rsync man page, it sounds like some combination of --delete-* and --exclude= should get us what we want, which is deletion of files that have disappeared, while retaining the .git folder.

[billyc]

rsync -vrultz --delete --exclude '.git*' [src] [dest]

seems to work