For each experiment evaluation, user can click on a button (or paste cmd in to terminal) that will cause the code to be exactly the code that was run at that time. The goal is perfectly reproducible experiments.
Implementation ideas
Every time the user runs an experiment, we hijack the user's git repository and make a secret commit. This commit hash is stored with the evaluation. Hitsave.io has a special git hosting service for each project which stores the secret commits.
All of this is configurable so the user can turn off code exfil if they don't want it.
research plan
[ ] how easy is it going to be to command git to make a secret commit without code-loss or messing up state / staging / stashing / submodules / etc etc etc
[ ] How do you deal with state / file snapshots?
[ ] can the 'secret commits' be hidden from the user?
Pure Effect:
For each experiment evaluation, user can click on a button (or paste cmd in to terminal) that will cause the code to be exactly the code that was run at that time. The goal is perfectly reproducible experiments.
Implementation ideas
Every time the user runs an experiment, we hijack the user's git repository and make a secret commit. This commit hash is stored with the evaluation. Hitsave.io has a special git hosting service for each project which stores the secret commits.
All of this is configurable so the user can turn off code exfil if they don't want it.
research plan