search

online-go / online-go.com

Source code for the Online-Go.com web interface
https://online-go.com/
GNU Affero General Public License v3.0
1.26k stars 346 forks source link

AI annotation inconsistency #1075

Closed benjimin closed 1 year ago

benjimin commented 4 years ago

I've noticed that in AI review, the blue circle is not always the greatest value among the numbers displayed.

This is inconsistent with the documentation (that although the green shades mean something else, the numbers are the impact on win probability, and the blue is the most optimal move).

Is this expected behaviour (that might warrant further clarification of the documentation), or is it a bug? Is there something I can inspect to know if the analysis data is strange or just the visualisation layer?

https://online-go.com/game/22649372

image

image

tested on: macOS Catalina, Chrome 80.0

flovo commented 4 years ago

Hi,

thank you for your report. I had a look at your game. The numbers seem to differ from your screenshots. I've identified move 15 for your first screenshot, and move 34 for your second screenshot.

The KataGo AI review I've found and used is: image move 15: image

move 34: image

There was also a LeelaZero run, but it matched even less with your screenshot.

The numbers I've seen are in an expected range for the top move. The move suggested by the AI hasn't to be the move with the highest winrate. The AI calculates a confidence estimation as well. The suggested move is then calculated using winrate and confidence. (The documentation fails to mention this)

Also in your first screenshot, the AI hasn't explored the played move at all (no green background). It could be just a blind spot of the AI.

Your second screenshot is strange. The suggested move has a much lower winrate than the other well explored move. But it looks much better now.

The current reviews are Supporter level 2 reviews. Does that match your Supporter level (when you made the screenshots)?

And can you please verify if you still have the numbers of your screenshot? Can you provide which AI review you are using if this is still the case?

If you are interested in the raw data, you can take a look at

https://online-go.com/api/v1/games/22649372/ai_reviews for the review_ids and https://online-go.com/termination-api/game/22649372/ai_review/3239722 for the raw data. What you're seeing on the goban is just a visualization of those data with only minor adjustments. It should either be in agreement with the raw data in all cases, or should be wrong for all cases.

The branches for each move are sorted by preference of the AI. Branch 0 is the move the AI would play, branch 1 is its second choice, and so on. Coordinates for each axis are 0..18 with 0-0 being the top left corner (A19).

JDNdeveloper commented 3 years ago

Just wanted to give this a bump. I've noticed this a lot in my games, where the optimal move shown is pretty much always "0", but if it's also the move I played then it has a non-zero win rate.

For example:

When it's NOT the move played: image

When it's the move played: image

Game for reference: https://online-go.com/game/29740115/7

github-actions[bot] commented 1 year ago

This issue has been marked stale and will be closed soon without further activity. To keep the issue open, please respond to the comment to keep the discussion going.

github-actions[bot] commented 1 year ago

This issue has been automatically closed due to inactivity. Please feel free to re-open it if the issue persists.