tensorflow / minigo

An open-source implementation of the AlphaGoZero algorithm

Apache License 2.0

3.47k stars 558 forks source link

Reproducability & Auditability #591

Open sethtroisi opened 5 years ago

sethtroisi commented 5 years ago

Some comments from offline discussion between Andrew & Seth

Launch things via scripts & flag files
record CLI & flag files into a directories
model table
run table
eval table
- [x] v1 was ratings.db in ratings.py
- [x] v2 is bigtable
- [x] Backfill script https://github.com/tensorflow/minigo/pull/776
- [] Run and verified backfill script ...

sethtroisi commented 5 years ago

Model registry

Models get a parent (which should also be in the registry)
save the commands invoked (python dual_net.py train ...)
save the master repo diff
any local code diff
- How to do this?

Results table

[x] Uses bigtable tag https://github.com/tensorflow/minigo/pull/590 to name a comparison
[x] Use launch_eval.py and record command (somewhere)
- First pass done in https://github.com/tensorflow/minigo/pull/601

Each entry should be

contained in a google cloud directory (probably in minigo-pub)
contain the information to reproduce / understand where it came from
force push code to a testing branch and create the url for diff between that branch and master commit

sethtroisi commented 4 years ago

919 and #893 are both related