Goreviewpartner (GRP) , automated review tool, now supports phoenixgo native engine !!

wonderingabout commented 5 years ago

The support is still incomplete, but it works !! Congratulations @wodesuck

GoReviewPartner is a famous sgf analysis tool

it can analyze all moves of a sgf (or only some moves), it can also analyze many sgf automatically in one run

The analysis runs on the background, you can do other things and wait until it is finished, then the generated .rsgf file containing many variations per move for every move can be viewed on the GRP "open .rsgf"

see : https://github.com/pnprog/goreviewpartner/pull/96

and : https://github.com/pnprog/goreviewpartner

grppgo1 grppgo2 grppgo3

prp1 prp2 prp3

prp4 ppp1

winrate curve difference

(part of the difference is winrate black x , white 100-x => yoyo in the graph)

native phoenixgo 3000 sims batch 4 400M tree size 64 max children tensorrt ON

ppp0c

LZ phoenixgo-orig -v 3200

lzphoenixgogrp

wonderingabout commented 5 years ago

example of moves analyzed by native PhoenixGo :

bb0 bb1 bb2 bb3 cc0 cc1 cc2 dd0 dd1 dd2 dd3

wonderingabout commented 5 years ago

compare it to LZ-phoenixgo-orig : https://github.com/alreadydone/lz/tree/phoenix-orig grp release 0.14.2

pnprog commented 5 years ago

Hi @wonderingabout ,

Can you share both RSGF files for me to check? I am not sure why this "yoyo" effect. If it was a consequence of "100 - x", the beginning of the graph (left part) would be different for both engines, but this is not the case. So it must me some other issue.

A "yoyo" effect can be normal, when both players keep failing to see (and play) a game reversing move. But in this case, both engines should have the same result.

wonderingabout commented 5 years ago

@pnprog

here it is :+1:

grp versions are different

grp cloned from github : http://www.mediafire.com/file/bubs1wi7x86zbk3/game2nativePhoenixGo.rsgf/file grp release 14.2 : http://www.mediafire.com/file/dtw8taad9hg2qeo/game2LZPG.rsgf/file

also one bad behaviour i noticed is its possible for white to play in an already played black stone !!

pnprog commented 5 years ago

Thanks,

I can confirm there are some issues with GRP there. For example, move 27 has two variations for o17, which obviously is wrong. The issue is most probably linked with GRP parsing all the log from PhoenixGo.

I will have to fix this before we start comparing both engines. There may be other issues to fix also :)

wonderingabout commented 5 years ago

we can use this game as a benchmark when you release some fixes

i'll rerun an analysis and we'll see if its better

pnprog commented 5 years ago

Maybe some "edge cases" that appear only when a lot of playout are performed: Maybe the order in which the variations are displayed is not consistent, and GRP got confused or something like this.

I will try to have my computer analyse that move 27 during the whole night, and maybe I will find something inside.

wonderingabout commented 5 years ago

take it easy :)

alreadydone commented 5 years ago

@godmoves (of HappyGo expertise) might want to tune in. (See also https://github.com/pnprog/goreviewpartner/pull/96)

pnprog commented 5 years ago

@wonderingabout

i'll rerun an analysis and we'll see if its better

I just pushed some changes on GRP repository. This should fix the issue we saw above, and also add information about number of playouts (in fact, number of visits). And also use less memory.

Could you have a try again now with that same game, so that we can compare the winrate curves? Don't hesitate to go with bigger tree size (maybe width=6, depth=10?) I am quite confident GRP should handle that properly now.

wonderingabout commented 5 years ago

@pnprog

ok will start it now then :+1:

but lets just start with 6-6 for starters :)

also one thing i noticed when i close grp is that i'm not in contributors list :) i dont mind much, but i think you would care more about it than me, am i right :)

wonderingabout commented 5 years ago

@pnprog

analysing

gtx1060 power capped at 75W for low temps :+1:

wonderingabout commented 5 years ago

also 2 : i'll increase sims to 3200 to have equal comparison with LZ-PhoenixGo

and also 3 : do i have to use --logtostderr in conf ?

pnprog commented 5 years ago

yes, logtostderr and verbose 1 are needed.

wonderingabout commented 5 years ago

i ask because i saw that you removed --logtostderr here : (i used it everytime though)

https://github.com/pnprog/goreviewpartner/blob/master/config.ini#L100-L104

since phoenixgo can be a bit hard to configure, can i suggest that we add 2 examples here (1 for tensorrt, and 1 for notensorrt)

[PhoenixGo-0]
profile = Example-tensorrt
command = /home/yourusername/PhoenixGo/bazel-bin/mcts/mcts_main
parameters = --gtp --config_path=/home/yourusername/PhoenixGo/etc/mcts_1gpu_grp.conf --logtostderr --v=1
timepermove = 120

profile = Example-notensorrt-linux
command = /home/yourusername/PhoenixGo/bazel-bin/mcts/mcts_main
parameters = --gtp --config_path=/home/yourusername/PhoenixGo/etc/mcts_1gpu_notensorrt_grp.conf --logtostderr --v=1
timepermove = 120

profile = Example-notensorrt-windows
command = C:\users\yourusername\Downloads\PhoenixGo\bin\mcts_main
parameters = --gtp --config_path C:\users\yourusername\Downloads\PhoenixGo\bin\mcts_main --logtostderr --v 1
timepermove = 120

i'm sure these examples will be helpful for interested people, because there are a lot of things to pay attention to

i'll also add 2 new config files optimized for grp on phoenixgo github :

remove time management to use fixed visits
remove all other time management options
mention that it is just an example
add the debug @alreadydone mentionned, with example of 6x6

also, i would like if you sent a PR to my branch, after all you really helped phoenixgo be better

maybe i can commit with your username and email ? (i never tried it before so i wonder if it will work with fake username and email)

wonderingabout commented 5 years ago

btw time where i am now :

grptime

pnprog commented 5 years ago

i ask because i saw that you removed --logtostderr here

I haven't think too much about this. On my own set up, I set the command directly to the start.sh file, that's why (and maybe that's not the best idea).

For the moment, the entry is just called "Test" or something, because there are still things I need to test. Like handicap games (apparently PhoenixGo support up to 5 stones), and if the GTP time_setting command can really be used (in that case, I will add an entry in the setting panel just like Leela).

When everything is more or less ok, I will create two entries (slow and fast) like the other bots.

So for the next release of GRP (hopefully around February 10th) I will package one download for PhoenixGo (for Windows) already configured, and the weights file included. And because everyone's hardware is different, there will be 3 downloads, one for GPU, one for CUDA's GPU, and one for CPU. So most of Windows users will be covered. For other users, I will detail in details in the documentation what settings are necessary to have GRP work with PhoenixGo.

Do you have pushed your mcts_1gpu_grp.conf files somewhere already?

wonderingabout commented 5 years ago

oh wow i'm so careless lol

it just finished but forgot to use the new config file, gotta redo the analysis again lol

wonderingabout commented 5 years ago

oh no actually i really used the right config file, my bad

wonderingabout commented 5 years ago

so here is the result :+1:

http://www.mediafire.com/file/dvp18509dtj5eru/game2v2.rsgf/file

graph2

still some things to fix :)

wonderingabout commented 5 years ago

handicap is a bad idea, zero bots are too strong, max playable is 2 (or 3 for 10k or weaker players, with custom config file)

i dont know about start.sh, but i like to have my own settings :) (especially on windows where you need special syntax)

pnprog commented 5 years ago

still some things to fix :)

Yeah :)

I'm moving the discussion back to GRP project, to avoid polluting PhoenixGo project with GRP's issues :)

wonderingabout commented 5 years ago

discussion continues here :

https://github.com/pnprog/goreviewpartner/issues/99

pnprog commented 5 years ago

also, i would like if you sent a PR to my branch, after all you really helped phoenixgo be better

Here it is: https://github.com/pnprog/PhoenixGo/tree/patch-3

In the context of GRP, because time is not critical (there is no risk of loosing a game on time) it is better to let the engine thinks more time if it can help it figure out better the best play. I do something similar for Pachi. In fact, I would go as far as increasing the time_factor up to 1.0 (0.3 is the default value).

I don't know how to PR to your branch, github only proposes me to PR to Tencent's master branch at this point.

By the way, I did not see a mcts_cpu_grp.conf file :) And that's a lot of modifications to PhoenixGo's project, that are related to an other project, so they may not be interested in it anyway :)

wonderingabout commented 5 years ago

also, i would like if you sent a PR to my branch, after all you really helped phoenixgo be better

Here it is: https://github.com/pnprog/PhoenixGo/tree/patch-3

great :+1: @pnprog

now you need to go to pull requests of my fork here :

https://github.com/wonderingabout/PhoenixGo/pulls

then create new pull request

compare :

wonderingabout/PhoenixGo - branch faqv2-bazel-master

with

pnprog/PhoenixGo - branch patch-3

i will merge you pull request into my branch when you create the pull request :+1:

wonderingabout commented 5 years ago

to answer your questions now, the grp files i created have time management all disabled (unlimited time), (all enable time options are set to 0), and timeouts set to 0 ms (infinite time)

infinite time :

timeout_ms_per_step: 0

unlimited time :

early_stop {
    enable: 0
    check_every_ms: 100
    sims_factor: 1.0
    sims_threshold: 2000
}
unstable_overtime {
    enable: 0
    time_factor: 0.3
}
behind_overtime {
    enable: 0
    act_threshold: 0.0
    time_factor: 0.3
}
time_control {
    enable: 0
    c_denom: 20
    c_maxply: 40
    reserved_time: 1.0
}

i forgot indeed the cpu version (and the cpu profile too, i thought it was included in notensorrt :laughing: ), can you use this opportunity to send it to me ?

pnprog commented 5 years ago

I can't find wonderingabout/PhoenixGo - branch faqv2-bazel-master in the menu Choose a Base Repository, and I have no clue why. I am really helpless about all those github related things :(

wonderingabout commented 5 years ago

@pnprog

yes github is not intuitive to understand at firs, but we all went through that :+1:

the problem you have is because you forked from Tencent/PhoenixGo, not from wonderingabout/PhoenixGo, so even though you managed to import my branch on your fork of Tencent/PhoenixGo, the automated pull requests point to Tencent/PhoenixGo where the faqv2-bazel branch doesnt exist

to fix that, i think you can write manually the repoowner/project e.g wonderingabout/PhoenixGo and find my branch in pull request

if not, then you can backup all your branches (download zip), then delete Tencent/PhoenixGo (in settings "dangerous zone" : https://help.github.com/articles/deleting-a-repository/ click to see screenshots)

then go to wonderingabout/PhoenixGo and fork it here "click fork" : https://github.com/wonderingabout/PhoenixGo

wonderingabout commented 5 years ago

i think we can close this issue now,

support questions and research can continue in : https://github.com/pnprog/goreviewpartner/issues/99

Tencent / PhoenixGo

Goreviewpartner (GRP) , automated review tool, now supports phoenixgo native engine !! #86