Distributed network testing protocol

[marcocalignano]

I wanted to open another Issue for this discussion. Here we can discuss the commands the server send to the clients and the answer the clients send back to the server. At the end we make a decision (I guess @gcp have the last word ;) ). In the meanwhile @roy7 and me can concentrate on writing the client/server code without having to change it too many times. This is my suggestion for the two commands

{
        "cmd":"match",
        "white_hash":"223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6",
        "black_hash":"92c658d7325fe38f0c8adbbb1444ed17afd891b9f208003c272547a7bcb87909",
        options {
              "playouts":1000,
              "resignation_percent":3,
              "required_client_version":5,
              "noise":false,
              "randomcnt":0,
              "version" : 1
       }
}

{
        "cmd":"selfplay",
        "hash":"223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6",
        options {
              "playouts":1000,
              "resignation_percent":3,
              "required_client_version":5,
              "noise":true,
              "randomcnt":30,
              "version" : 1
        }
    }

[marcocalignano]

And this is the suggestion for the two answers:

-F winnerhash=aaaa
-F loserhash=bbbb
-F clientversion=6
-F winnercolor=black
-F movescount=321
-F score=B+45
-F optionver=1
http://zero-test.sjeng.org/submit-match

-F networkhash=aaaa
-F clientversion=6
-F sgf=@file
-F trainingdata=@data_file);
http://zero-test.sjeng.org/submit

[roy7]

Some people have requested we upload the sgf files for matches. Could you upload those with the game result? I can change things on my end to move options into the options.* wrapper. I'm still at work for 4 more hours so it won't be immediate. :)

[marcocalignano]

@roy7 I'd like to test other stuff first but if you need it I'll do it

[roy7]

You don't need to support it yet. Do it when everything else works. :)

[sethtroisi]

consider replacing "optionver=1" with a hash of the options calculated on the client side.

[roy7]

On the test server, /get-task now uses the options.{} wrapper for all the options like you specified.

What does version=1 relate to? Version of this protocol?

[roy7]

In fact I'll move this to live server as well, since no one is running your code yet.

[roy7]

@marcocalignano Do you want to edit your specification above to have a -F sgf=@file for /submit-match as well? Then I'll program the server side to require and expect a compressed sgf file the same as /submit.

[marcocalignano]

What does version=1 relate to? Version of this protocol?

this is a version for the options object. I thought that maybe we want to do some experiments with the same networks but with different options, so you need a way to identify which submitted matches were play with one set of option or the other. The two networks are not enough anymore.

For the option -F sgf=@file I'd like to wait for more inputs discussion maybe suggestions from @gcp

[marcocalignano]

@sethtroisi Why an hash should be better than a version number?

[roy7]

I see the point of storing the version (or hash or whatever we do), so like we could do some comparisons between two networks at 1000 playouts and 1600 playouts if we wanted and tell them apart with the version #.

I'm guessing the reason he suggested a hash is so we don't manually need a version #. If we run the same match with new options, the hash would change so they could all be grouped by hash automatically.

[marcocalignano]

Hash help not to make mistakes (two versions that are the same) but with the hash is more complicated to search in the database (I guess). But there are smarter people than me in SQL that would say 'easy peasy' ;)

[roy7]

I'm having trouble deciding on a database structure for the matches on the server, but actual match game uploading should work now and be stored. I don't have the code in place to choose when to request matches yet though since I've been having a mental roadblock on the data structure. Compressed sgf is required when uploading match games.

[isty2e]

A tiny suggestion, but what about "options_version" instead of "version" for clarification?

[marcocalignano]

@isty2e version is inside the object called option

[isty2e]

Some might consider it self-explanatory or obvious, but there can also people confused with it at the first sight like me or roy7. Slightly lengthening key wouldn't harm too much, I guess. Of course if you don't like to change it, well it's your choice.

[marcocalignano]

So the current proposal is:

{
   "cmd" : "match",
   "white_hash" : "223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6",
   "black_hash" : "92c658d7325fe38f0c8adbbb1444ed17afd891b9f208003c272547a7bcb87909",
   "options_hash" : "c2e3"
   "options" : {
       "playouts" : 1000,
       "resignation_percent" : 3,
       "required_client_version" : 5,
       "noise" : "false",
       "randomcnt" : 0
    }
}

{
   "cmd" : "selfplay",
   "hash" : "223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6",
   "options_hash" : "ee21"
   "options" : {
       "playouts" : 1000,
       "resignation_percent" : 3,
       "required_client_version" : 5,
       "noise" : "true",
       "randomcnt" : 30
    }
}

and

-F winnerhash=223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6
-F loserhash=92c658d7325fe38f0c8adbbb1444ed17afd891b9f208003c272547a7bcb87909
-F clientversion=6
-F winnercolor=black
-F movescount=321
-F score=B+45
-F options_hash=c2e3
-F sgf=@file
http://zero-test.sjeng.org/submit-match

-F networkhash=223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6
-F clientversion=6
-F options_hash=ee21
-F sgf=@file
-F trainingdata=@data_file
http://zero-test.sjeng.org/submit

[roy7]

Is options_hash something the server will calculate and send, and you'll just echo it back when reporting in?

[marcocalignano]

@roy7 yes! So the server can connect the hash to the options that (I guess) are saved in a different DB table.

[roy7]

Ok cool that'll be helpful. In fact, I can then more easily support multiple matches in the match database with different options as well.

[roy7]

Ok options_hash is supported now. If I'm not mistaken, all reporting back to server is done and the client requests are both handled correctly. The open task for me tonight is finish up /get-task when it sends out a match, to actually generate those off my database of pending matches. Then behind the scenes I'll add some helpful functions for gcp to easily add new matches to the list when he wants them run.

[marcocalignano]

@roy7 do you need to leave my client running to test your server script?

[roy7]

Probably not required, but no harm if you do want to. Since you'll just do self-play games for now anyway like normal clients. :)

[marcocalignano]

but if I push on the test server you can change the commands and see if it works. For example if you do test commands the hash has to alternate white black so you can test it

[marcocalignano]

One thing I have noticed is that required_client_version goes outside the option object. like that:

{
   "cmd" : "selfplay",
   "hash" : "223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6",
   "options_hash" : "ee21",
   "required_client_version" : 5,
   "options" : {
       "playouts" : 1000,
       "resignation_percent" : 3,
       "noise" : "true",
       "randomcnt" : 30
    }

}

[roy7]

I put it in options at the moment. You want me to move it outside, right? Yes you are welcome to run your autogtp on the test server for now. :)

[marcocalignano]

yes please put it outside so I give it a test! The sgf has to be gziped right?

[roy7]

It's outside now.

sgf should be gzip'ed yes. Thanks.

[marcocalignano]

Client is running

[marcocalignano]

You should have gotten two self played games

[marcocalignano]

If that is true you can try with a test match

[roy7]

Ok I see from your games I wasn't storing options_hash, I am now. I'll let 1 more game come in to check it then I'll change you to getting matches instead of selfplays.

[roy7]

Test server now serving matches. Your normal game submissions looked good.

[marcocalignano]

what about the test match submission (any other spelling mistake?)

[roy7]

@marcocalignano I see task requests but no test match submissions at all. Hmm. You see no response text from curl on your end?

[marcocalignano]

Oh yes lost in the debug output Upload failed. Curl Exit code: 26 my command line is curl -F winnerhash=92c658d7325fe38f0c8adbbb1444ed17afd891b9f208003c272547a7bcb87909 -F loserhash=223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6 -F clientversion=6 -F winnercolor=black -F movescount=266 -F score=B+0.5 -F options_hash=3e4e92 -F sgf=@f285bc656a9a4d0084f38c587f74fd8e.gz http://zero-test.sjeng.org/submit-match

[marcocalignano]

OK @roy7 bug on retreiving the sgf file name let's see how it does now.

[roy7]

I wish it'd print the result text from the server, if any, since I do return clear error text messages. I'm logging any call to /submit-match at all though, and I'm not seeing it being called. So it might be failing before it even goes out. Do you need to escape the @ on your command line? I normally wrap that in quotes.

[marcocalignano]

I believe it failed on my side because it did not find the file I wanted to send

[roy7]

Yeah curl error 26 is 'can't open file'.

[marcocalignano]

Now?

[roy7]

{"winnerhash":"92c658d7325fe38f0c8adbbb1444ed17afd891b9f208003c272547a7bcb87909","loserhash":"223737476718d58a4a5b0f317a1eeeb4b38f0c06af5ab65cb9d76d68d9abadb6","clientversion":"6","winnercolor":"black","movescount":"200","score":"B+4.5","options_hash":"3e4e92"}

[roy7]

Although it didn't process it. Checking.

[roy7]

It seems you have all the fields although I didn't print out if you included a full sgf file. What was the response on your side to that curl command?

[marcocalignano]

I didn't print it out. It will wit ht e next game!

[roy7]

Found a bug that I wasn't storing the sgf game itself, fixed. Hopefully your next submit goes smooth!

[roy7]

@marcocalignano Score was "" on your most recent submission btw.

[roy7]

@marcocalignano Got your match, and it saved fine. I had a mismatch between the dummy match and the database's match info, so I fixed that too just now. If you upload 1 more it should update everything on my end properly now!

[marcocalignano]

OHHHHHH I found a bug that a lot of people thought was something else!!!!!

[roy7]

Latest upload worked as expected, both database tables updated properly. I think it's working.