We need a document that details all the steps that need to be taken to add a new line to the leaderboard (= run a new model, or run a model again).
See the versioning_running.md planning document: Changes in the set of games constitute an increment in the major version number of the benchmark / leaderboard; changes to the instances for a set of game constitute an increment in the minior version.
How to make sure that all games and game instances (i.e., the exact version number) match that of the leaderboard to which the results are to be added? (That is, what is to check? The commit hash? Should we have a version.txt file in each game directory? How are the instances for a game identified?)
What exactly are the steps for taking the results for the given model (and from where?) and adding it to the results file (a csv?) that constitutes the current leaderboard?
We need a document that details all the steps that need to be taken to add a new line to the leaderboard (= run a new model, or run a model again).
See the
versioning_running.md
planning document: Changes in the set of games constitute an increment in the major version number of the benchmark / leaderboard; changes to the instances for a set of game constitute an increment in the minior version.