search

SabakiHQ / Sabaki

An elegant Go board and SGF editor for a more civilized age.
https://sabaki.yichuanshen.de/
MIT License
2.41k stars 377 forks source link

Perform full game analysis #487

Open vascopimenta opened 5 years ago

vascopimenta commented 5 years ago

Apologies if this is the wrong place to suggest features.

After some trial and error, I've finally been able to have Leela Zero working with Sabaki on macOS. Kudos to the developers for all the work put into supporting this.

As it stands, if one wants to save LZ suggested variations, they have to be manually added (right-click on a given spot on the heatmap -> Add variation). The resulting file can then be saved, and these variations/branches show up when one opens the sgf file. Also, the winrate graph is retained, even though at this stage it seems buggy.

I believe it would be a very interesting feature (especially to those of us relying on CPU-only versions of LZ) to be able to automatically have LZ attached and perform a full analysis of the game, let's say, providing a maximum thinking time per move, and have the n highest visited locations saved as branches for each position. Another interesting option would be to store those branches only if the variation in estimated winrate after a given move varied for more than x%. Ideally, also the heatmap, winrate %, number of visits would be saved and shown as well.

This would basically give you an sgf file with the analysis of your game, showing where you and your opponent made the most important mistakes, and would IMHO be a very useful tool for improving at Go. I imagine these features being implemented in a different mode/dialog box - say, Tools -> Game Analysis, where you would select engine to attach, thinking time, the above described criteria, etc., and then you would press start and wait for it to run over the whole game and save the output over the existing sgf file (or a new one, as an option).

Sorry for the not very focused suggestion, I hope the central ideas come across. Feel free to ask for clarifications, if you find the idea interesting, and thank you for Sabaki.

yishn commented 5 years ago

Hi, thanks for the suggestion. FYI there's currently a separate tool for doing just that, called GoReviewPartner by @pnprog. You can feed it any sgf file and it will create an rsgf file which you can open with Sabaki.

Doing a full game analysis will take a looong time, during which you cannot edit the game tree. You might as well use an external tool for that. Nevertheless, I'm considering such a feature for the future for Sabaki as well.

Also, the winrate graph is retained, even though at this stage it seems buggy.

Please open a separate issue describing why it's buggy.

vascopimenta commented 5 years ago

Please open a separate issue describing why it's buggy.

Will do. Will first try to find out reproducible steps to provoke the issue (suddenly the red bars disappear, and only the white graph remains visible).

vascopimenta commented 5 years ago

You can feed it any sgf file and it will create an rsgf file which you can open with Sabaki.

Sabaki does not open the rsgf files output by GoReviewPartner. Should I be able to open them?

yishn commented 5 years ago

Since rsgf files are just big sgf files, Sabaki should be able to open them. Are you getting some error?

pnprog commented 5 years ago

Hi @yishn ,

Nevertheless, I'm considering such a feature for the future for Sabaki as well.

Since rsgf files are just big sgf files, Sabaki should be able to open them.

If you are interested, we could collaborate a little on this topic. RSGF files are simply SGF files with a few non standards extensions to store data provided by Leela, AQ... and I am opened to new idea to improve this.

So maybe we could work out a standard for the RSGF files that says exactly what are the format and meaning of the extensions. This way, a RSGF file created by GRP can be displayed in Sabaki and vice versa.

A simple way would be to create a dedicated github repository to store a description of the RSGF extensions, and suggestions by users or software developers can be discussed through issues and pull request.

What do you think?

yishn commented 5 years ago

This sounds great! I would be interested in what kind of extensions you have for rsgf. Is there any reason why these files have their own extension instead of plain old sgf?

pnprog commented 5 years ago

Sorry, my wording was wrong, I should I said:

RSGF files are simply SGF files with a few non standards properties (not extensions) to store data provided by Leela, AQ...

Some of the properties used are:

So yes, it is still plain old SGF that can be read by any SGF editor. But I think it could probably be improved.

vascopimenta commented 5 years ago

Since rsgf files are just big sgf files, Sabaki should be able to open them. Are you getting some error?

I can open them, if I rename them to .sgf. Otherwise I get a "This file is unreadable" message. This happens both while trying to open the file through the open dialog box in Sabaki, or if right-clicking on the file, choosing open with, and then selecting Sabaki. I'm on mac OS. I've since realised that GRP has an option to convert rsgf into sgf files.

W.r.t. Sabaki and GRP-like functionality, Sabaki has a much nicer interface, and the visualization of good variations, with numbers on a heatmap, and automatic playout of variations on "on mouse over" is IMHO much preferrable to GRP. Also, when saving branches/variations calculated by LZ directly on Sabaki, additional properties are saved, such as marking the variations "good move", etc.

Ability to resize the graph window and some way of displaying the table containing number of playouts, win probability, etc. would be great nice-to-haves.

Also, as a non-coder with limited software skills, I also wonder if there is any practical and/or philosophical obstacle to including LZ binaries with Sabaki, instead of having to have users compile it. I believe this would much widen the audience that could benefit from these tools. I realize this may be a bit off-topic, feel free to edit as you see fit.

Thank you both @yishn and @pnprog for your great tools.

yishn commented 5 years ago

I can open them, if I rename them to .sgf. Otherwise I get a "This file is unreadable" message.

This was a bug; Sabaki should use the SGF loader if file extension is not recognized. I've fixed it in master.

Ability to resize the graph window

Thanks to @dbosst, this is coming in the next version, too.

I also wonder if there is any practical and/or philosophical obstacle to including LZ binaries with Sabaki, instead of having to have users compile it.

Philosophical reason: Sabaki's philosophy is to work with any engine you want, configured to your liking. This also implies getting the engine binaries yourself.

Legal reason: Leela Zero is released under a GPL license, but Sabaki is released under the more tolerant (and incompatible) MIT license. Since it's a bit unclear whether a software that includes GPL binaries are considered derivative work or not, common advice is to not include them.

Practical reason: I don't want to compile the source for an engine for three different platforms on every new engine version, or release a new Sabaki version on every new engine version. Making the user manage their own engines will save me a lot of work. It's already a miracle that I can build binaries for all three platforms thanks to the amazing Electron ecosystem.

vascopimenta commented 5 years ago

Great, new enhancements always in the pipeline. :-)

Perhaps explicitly adding rgsf to the extensions that would open in Sabaki by default? One can always add it later, of course.

Given that I have gone through the trouble of doing it, I could volunteer (and perhaps others could too) the binaries, not necessarily always the latest versions, but I understand the issue with MIT/GPL.

Phelanpt commented 5 years ago

Maybe this is not the best spot for this, but I'm not quite sure what to name a new issue: The last two versions of Sabaki seem to have broken the way I used it to run a full analysis on a game. I used to turn on both Analysis mode and Autoplay mode, and just let it run, but currently, after I start Autoplay by clicking the "play" button, Sabaki goes back to "stopped" mode on autoplay after a while. I haven't been able to pinpoint anything that might be causing this, so I'm sorry if this is not very helpful.

To reproduce: Attach an engine to a player, wait for it to load. Toggle analysis mode on. Toggle autoplay on, then click the "play" button. It seems to revert back to "stopped" status after a little bit, and stay that way. :/

yishn commented 5 years ago

@Phelanpt I've created a new issue for that: #571

kevinsung commented 3 years ago

A good first step for this issue would be to let Sabaki recognize non-standard SGF properties that would be added by the analysis program. I've been working on this, and have been using the following property names, which are open to change:

The idea is that if Sabaki opens an SGF file with these properties, it can display this analysis information. If an engine analysis session is started, the information from that session would override the information from the SGF for the duration of the session.

kevinsung commented 3 years ago

Sorry, my wording was wrong, I should I said:

RSGF files are simply SGF files with a few non standards properties (not extensions) to store data provided by Leela, AQ...

Some of the properties used are:

* BWWR[...] => black/white win probability for a variation

* BKMV[...] => Book move

* PNV[...] => Policy network value for a variation

* MCWR[...] => Monte Carlo win probability for a variation

* VNWR[...] => Value network black/white win probability for a variation

* PLYO[...] => Number of playouts used to estimate a variation

So yes, it is still plain old SGF that can be read by any SGF editor. But I think it could probably be improved.

@pnprog What does BWWR actually mean? Is it the winrate for the current player, whichever color that may be? Also, which one of your winrate properties would you consider to be "the" winrate that should be displayed in Sabaki's winrate graph?

The reason I'm asking is that I'm working on some PRs to let Sabaki recognize non-standard SGF properties used by analysis programs, and it would be good to use or develop some kind of standard that can be shared between tools.