Performance evaluation using ELO ratings

[AbdBarho]

Current Plan (28/02/2018):

For the task of evaluating the performance of our current model we have planned the following:

• Find "anchor" models:

  • These will play against our model to determine the approximate ELO of our model.
  • Currently we are trying to implement a simple match system between our model and GnuGo
  • A total of three anchor models is planned to be used (including a version of our model with totally random weights).

• Let the models play to find the ELO Rating:

  • No concrete parameters are currently known for the matches, ex: number of rounds, start ratings, evaluation time for each move, etc..
  • We will get to that as soon as we have a match making system

• Implement the ability for our model to play against other versions of itself:

  • This will be used in the training phase to determine if the new model version is better than the last one.
  • This Feature is needed by other sub projects in the Argos-Zero projekt (and it may be implemented earlier).

This is a rough sketch of our plan for the coming two weeks and it may change.

[AbdBarho]

Update (28/02/2018):

• Found two possible anchors for evaluating our model, Leela (ELO:2400) and GnuGo (ELO~1550) are most likely candidates.
• Tried IPC between GTP engines using C++ popen: it was not the optimal solution as the stdout value will only be read when the process terminates.
• Tried using Boost.Process which had to be compiled from source, it was too big and not easily available so we discarded the idea.
• Finally implemented a simple IPC pipe communication using Python's subprocess module, example code:

  from subprocess import Popen, PIPE
  p = Popen(["gnugo","--mode", "gtp"],stdin=PIPE,stdout=PIPE)
  p.stdin.write('showboard\n')
  line = ""
  rd = p.stdout.readline()
  while( not rd == "\n"):
  line += rd + "\n"
  rd = p.stdout.readline()
  print line  # prints the board

[AbdBarho]

Update (01/03/2018):

• Implemented a prototype of the model evaluations system, currently under /python/Model Evaluation
• Researched Glicko-2 as a possible rating system, which should be better than the standard ELO system

[AbdBarho]

Update (02/03/2018):

• Updated the model evaluation system:
  • it uses the Glicko 2 Rating System
  • added some predefined ratings in the python/Model Evaluation/ratings.txt file, which will be used for estimating the rating (when playing against different models)
  • it now accepts parameter both in the command line and directly in the constructor:
  • path and arguments for both players are required
  • numMatches: number of matches to play (default 1)
  • batchSize: how many matches will be run before each Glicko score update (default 1)
  • verbose: should show what is happening (default True)
• Added test examples:
  • test9x9.py: runs a simple match between the random version of our model vs GnuGo with a board size of 9 x 9
  • test19x19.py: runs a match between a our trained model and GnuGo on a 19x19 board
  • the precompiled versions are not uploaded to the repository because of their size (10+ MB)
  • to run the test:
    • you have to have GnuGo installed: apt-get install gnugo
    • you have to have a precompiled version of Argos-zero with the test parameters, it would be better to have it in the same folder as the test.py file
    • open test#x#.py and updated the launch parameters of Argos-zero (change the array, so that it runs the precompiled executable, ex: ["./gtp"].
    • you can also define other parameters to be used in the call: ["./gtp", "--boardsize","9", "--komi","6"], note that all parameters should be strings.
    • finally run using python test#x#.py

[AbdBarho]

Update (05/03/2018):

• Added another test example:
  • Play a match against Pachi on a 9x9 board
  • to test you need to download and compile Pachi yourself, the compiled binary will simply be called pachi.
  • using CDNN is optional and does not affect the test case, however using CDNN does make Pachi "smarter".
  • move to the compiled pachi binary file to the folder of test9x9_pachi.py and run python test9x9_pachi.py.
• Found a bug in the (implementation of) Glicko2 rating system:
  • rating changes do not make much sense.
  • we are not really sure if its a bug or a feature, the outcome however is not reasonable.
  • we are going to analyse the system more and then readjust and change what needs to be changed.
  • in the extreme case, we might reconsider using the standard ELO system.
• Updated the match system:
  • Finding winner when both players pass:
  • the old system used the GnuGo gtp command final_score to get the winner, which has led in some cases to complete playing the game until the end, which took a long a time.
  • we have considered using another Referee ("Pachi" to be more exact), however it has the same problem as GnuGo.
  • the solution was in the documentation of GnuGo: estimate_score; since the match is already done, the estimated score is the actual end score.
  • added the mode parameter to RatingSystem and Referee to use the right parameters when testing.
  • added rating of Pachi to the ratings.txt file, which should give a better estimate when testing.

[Verversl]

Update (06/03/2018):

• Umstieg vom Glicko2-Rating-System zurück zu dem klassischen Elo-System, da sich das Glicko2 System wie nicht gewünscht verhielt (was wohl an der sehr geringen Playerpool liegt). Die entsprechenden Anpassungen und Implementierungen haben länger gedauert als geplant und uns fast den ganzen Tag gekostet.

• Anpassung an die Änderungen im Masterbranch (config system)

• Bugfixes (Falls ein anderers Programm als Argos in einem Match aufgab führte das zu einem Error aufgrund eines Fehlers in der Logik des Programms. Dies war uns erst jetzt aufgefallen, da Argos zum ersten Mal im Match gegen ein anderes Programm gewann)

[Verversl]

Update (07/03/2018):

• Aktualisierung des Rating-Systems:
  • Jetzt verwendet unser Rating-System die neue Klasse "RatingManager", die alles was mit Bewertung zu tun hat steuert
  • Neue Parameter für die Ratings-Datei Name und Pfad
  • Diese Parameter werden von der neuen Klasse "RatingManager" verwendet
  • Nach NumMatches Mal werden die Ratings aktualisiert
  • NumMatches ist auch ein Parameter
  • Parameter-Parser akzeptiert jetzt die neu definierten Parameter
    
    
• Neue Klasse "RantingManager":
  • Elo-Ratings werden aus der Textdatei ausgelesen
  • Nach einem bestimmten Batch an gespielten Matches werden die aktualisierten Ratings in eine neue Datei gespeichert
  • Jede neu erstellte Datei enthält zudem einen "Timestamp" in der ersten Zeile um die zeitlichen Abläufe und chronologische Reihenfolge nachvollziehen zu können.
  • Da jeweils nur aus der neusten Datei gelesen wird, werden auch alle unangetasteten Ratings in der neuen Datei gespeichert
  • Wenn der Unterschied der Elo Zahlen ziemlich groß ist ändern sich die Elo-Zahlen bei einem Sieg des deutlich stärkeren überhaupt nicht
    
    
• Struktur der Ratings-Datei:
  • Alle Zeilen, die nicht mit öffnenden eckigen Klammern [ anfangen werden ignoriert
  • als UUID wird ein Array von Kommandozeilen-Parametern des Models verwendet (z.B. ['gnugo', '--mode', 'gtp', '--boardsize', '9'])
  • In der nächsten Zeile muss ein Array, das das Elo-Rating und den k-Faktor des entsprechenden Models enthält stehen (z.B. [1550, 20])
  • Wenn die nächste Zeile die Parameter nicht enthält wird das Rating des Models komplett ignoriert
  • Falls ein Modell in der Ratings-Datei nicht definiert ist oder ignoriert wurde erhält dieses die Default-Parameter: [1000,40]
  • Wenn das Rating manuell hinzugefügt werden soll, sollte die UUID so berechnet werden str(["param1", "param2", ...]), geprintet und dann kopiert

[AbdBarho]

Update (08/03/2018):

• Minor change to Match System:
  • the returned number of wins are now in a dictionary instead of an array
  • the keys for the dictionary are the stringified model parameters
  • we also had to updated Rating System to adapt to this change
    
    
• Updated the Test files:
  • since the new config update the log file is no longer a required parameter
  • this parameter has been removed from both test files test9x9_pachi.py and test9x9_gnugo.py
  • the ratings in the file /python/Model Evaluation/rating/ratings0 hast also been updated
    
    
• Added the EstimateElo function:
  • this functions runs a couple of matches between the model defined in the parameter and a list of predefined model with know ELOs
  • the list of predefined model contains (in the time of writing):
  • GnuGo (1550)
  • Pachi (2000) (Pachi has actually a rating of 2700 or 2d, but since the test will be run on a normal machine with standard performance, we decided to decrease the rating to a reasonable amount, 2000 is still high)
  • Random untrained version of our model (1000) ( we choose this number as a lower bound for the rating)
  • to update the rating of the models manually, you can change the values in the /python/Model Evaluation/rating/ratings# file, where # is the highest number found (the system will read the ratings from the last file)
  • the results of the function are still kind of questionable, since we only have 3 models to play against. to have a better estimation we need to have a lot more models
  • it should also be taken into consideration that a lot of matches have to be played to find a good ELO estimation