LeelaChessZero / lc0

The rewritten engine, originally for tensorflow. Now all other backends have been ported here.
GNU General Public License v3.0
2.45k stars 529 forks source link

Blunder collection of recent T60 based MLH nets #1403

Closed Naphthalin closed 2 years ago

Naphthalin commented 4 years ago

CCC prematch Lc0 - SF -- SF NNUE, game 202:

[Event "Pre-Match Preview (1|1)"]
[Site "?"]
[Date "2020.07.20"]
[Round "1"]
[White "Stockfish"]
[Black "Lc0"]
[Result "1-0"]

1. b3 d5 2. Nf3 Bf5 3. Bb2 h6 4. d3 Nf6 5. g3 e6 6. Bg2 Be7 7. O-O O-O 8. Nbd2
a5 9. Re1 c6 10. e4 Bh7 11. a3 Nbd7 12. Qe2 Qc7 13. a4 Rfd8 14. Rec1 Rac8 15.
exd5 Nxd5 16. Nc4 Bb4 17. c3 Be7 18. d4 Bf5 19. h3 h5 20. Ne3 Nxe3 21. Qxe3 c5
22. c4 Bf6 23. Rd1 Bg6 24. Rd2 cxd4 25. Bxd4 Nc5 26. Rad1 b6 27. Bxf6 Rxd2 28.
Qxd2 gxf6 29. Nh4 Bh7 30. Qh6 Rd8 31. Re1 Qe7 32. Qxh5 Qd6 33. Bf1 Kg7 34. Qg4+
Kf8 35. Qh5 Kg8 36. Ng2 Bg6 37. Qf3 Kg7 38. h4 Qd2 39. h5 Bf5 40. g4 Bh7 41. Re3
Qd4 42. Qc6 Ne4 43. Ne1 Ng5 44. Qc7 e5 45. Bg2 Rd7 46. Qc6 Rd6 47. Qa8 Rd8 48.
Qc6 e4 49. Qc7 Rd7 50. Qb8 Rd6 51. Nc2 Qd2 52. Ne1 Qd4 53. Kf1 Rd8 54. Qg3 Qb2
55. Re2 Qc1 56. Rc2 Qb1 57. Re2 Rd1 58. Qh2 Kg8 59. Re3 Rd2 60. h6 Bg6 61. Bxe4
Nxe4 62. Qb8+ Kh7 63. Qf8 Rxf2+ 64. Kg1 Rg2+ 65. Kxg2 Qb2+ 66. Kh1 Nf2+ 67. Kg1
f5 68. gxf5 Bh5 69. Rg3 Ng4 70. Rxg4 Bxg4 71. Qxf7+ Kh8 72. Qe8+ Kh7 73. Qg6+
Kh8 74. f6 Qd4+ 75. Kh1 Qd7 76. f7 Bf3+ 77. Nxf3 Qd1+ 78. Kh2 Qe2+ 79. Kg3 Qg2+
80. Kxg2 b5 81. f8=Q# 1-0

at move 59, Lc0 sv-t60-4175 (with 30k nodes) played 59. ... Rd2 in the position 6k1/5p1b/1p3p2/p5nP/P1P1p1P1/1P2R3/5PBQ/1q1rNK2 b - - 21 59, which lost the game.

Analysis with T70 shows that more time helps with finding the correct response 60. h6, so it might simply be a blunder under time pressure, but still an interesting position for future nets.

Naphthalin commented 4 years ago

CCC Lc0 - SF - SF NNUE match, game 74:

[Event "More Chesse-Play: Stockfish vs Lc0 vs Stockfish+NNUE (3|2)"]
[Site "?"]
[Date "2020.07.23"]
[Round "1"]
[White "Lc0"]
[Black "Stockfish"]
[Result "0-1"]

1. c4 c5 2. Nf3 Nf6 3. Nc3 Nc6 4. g3 d5 5. d4 cxd4 6. Nxd4 e5 7. Nxc6 bxc6 8.
Bg2 Rb8 9. Bg5 Rxb2 10. O-O Qa5 11. Qc1 Rb4 12. cxd5 Rc4 13. Qb1 Be7 14. d6 Bxd6
15. Qd3 Rd4 16. Bxc6+ Kf8 17. Qe3 h6 18. Bxf6 gxf6 19. Nb5 Rc4 20. Nxd6 Rxc6 21.
Nxc8 Rxc8 22. Qf3 Kg7 23. Qg4+ Kh7 24. Rac1 Qa6 25. Rfd1 Rhg8 26. Qf5+ Kg7 27.
Kg2 Rxc1 28. Rxc1 Qxa2 29. Qg4+ Kh7 30. Qf3 a5 31. Rc6 Rg6 32. Qd3 f5 33. Rc5
Qe6 34. Rxa5 Qc6+ 35. Kg1 Qc1+ 36. Kg2 Qc6+ 37. Rd5 Kg7 38. f3 Re6 39. Kh3 Qc1
40. Kg2 f4 41. Qe4 Qc3 42. Rd7 Qc6 43. Rd5 Qc3 44. Rd8 Qe1 45. g4 Qa5 46. Rd7
Qb5 47. Rd8 Qa5 48. Rd7 Qb5 49. Rc7 Qb6 50. Rc8 Qa5 51. Rb8 Qd2 52. Rc8 Qe1 53.
Qd3 Qa5 54. Qe4 Qb5 55. Rc7 Qb6 56. Rc8 Qa6 57. Rc5 Qa1 58. Rd5 Qc3 59. Rd7 Qc6
60. Rd5 Qc3 61. Rb5 Qa1 62. Rb8 Qd1 63. Rc8 Qd2 64. Rb8 Qd1 65. Rc8 Qe1 66. Rc7
Qd2 67. Rb7 Qd8 68. Rb5 Qc7 69. Qd3 Qc6 70. Rd5 Re8 71. Rd6 Qc7 72. Qa6 Ra8 73.
Rg6+ fxg6 74. Qxa8 Qe7 75. Kf1 h5 76. gxh5 gxh5 77. Qc6 Qf7 78. Qd6 Qf6 79. Qd1
Kg6 80. Kg2 Qg5+ 81. Kh1 Qe7 82. Qg1+ Kf7 83. Qb1 Kg7 84. Qd1 Kh6 85. Qd5 Qf6
86. Qc5 Kg5 87. Qg1+ Kh6 88. Qc5 Qe6 89. Qf8+ Kh7 90. Qd8 Kg6 91. Qd3+ Qf5 92.
Qd6+ Qf6 93. Qd5 Kg5 94. Qc5 Kh6 95. Qd5 Kg5 96. Qc5 Kh6 97. Qc1 Qg6 98. Qc7
Qb1+ 99. Kg2 Qf5 100. Qb8 Qg5+ 101. Kh1 Qf6 102. Qc7 Qe6 103. Qb8 Kg5 104. Qc7
Kg6 105. Kg2 Kg5 106. Kh1 Qf5 107. Qd8+ Qf6 108. Qd1 Qb6 109. Kg2 Qc7 110. Kh3
Qb6 111. Qd5 Qf6 112. Qg8+ Kh6 113. Qc8 Kg6 114. Qc3 Qe6+ 115. Kg2 Kf6 116. Qd3
Kg5 117. Qd2 Kg6 118. Qd8 e4 119. Qd2 e3 120. Qd8 Qf5 121. Kh1 Kh6 122. Qe7 Qb1+
123. Kg2 Qf5 124. Kh1 Kg6 125. Qd8 Qf6 126. Qd5 Kh6 127. Qc5 Kg6 128. Qd5 Kh6
129. Qc5 Qd8 130. Qc4 Kg7 131. Kg2 Qh4 132. Qd4+ Kg6 133. Qb6+ Kf7 134. Qb3+ Ke7
135. Qb4+ Ke8 136. Qb8+ Kd7 137. Qb5+ Kc7 138. Qa5+ Kc6 139. Qc3+ Kb5 140. Qb3+
Kc5 141. Qc3+ Kd5 142. Qd3+ Kc5 143. Qc3+ Kb5 144. Qb3+ Ka5 145. Qa3+ Kb5 146.
Qb2+ Kc4 147. Qc2+ Kb4 148. Qb2+ Kc4 149. Qa3 Qg5+ 150. Kf1 Qf5 151. Qa6+ Kb3
152. Qb7+ Kc2 153. Qe7 Qd5 154. Qc7+ Kb2 155. Qg7+ Kb3 156. Qa1 Kc2 157. Qa4+
Kb2 158. Qb4+ Qb3 159. Qd4+ Kb1 160. h4 Qc2 161. Qb4+ Kc1 162. Qa3+ Qb2 163.
Qc5+ Kd1 164. Qc4 Qc2 165. Qa6 Qb1 166. Qg6 Kc1 167. Qd6 Kc2+ 168. Kg2 Qd1 169.
Qxf4 Kd2 170. Qf6 Qxe2+ 171. Kg3 Qe1+ 172. Kg2 Qf2+ 173. Kh3 Qg1 174. Qb2+ Ke1
175. Qb6 Kf2 176. Qb2+ e2 177. Qd4+ Kf1 178. Qxg1+ Kxg1 179. Kg3 e1=Q+ 180. Kf4
Qe8 181. Kf5 Kg2 182. Kf6 Kxf3 183. Kg7 Qe7+ 184. Kg6 Kg4 185. Kh6 Kf5 186. Kxh5
Qh7# 0-1

at move 151, Lc0 sv-t60-4175 (with 90k nodes) played 151. Qa6+ in the position 8/8/8/5q1p/2k2p2/Q3pP2/4P2P/5K2 w - - 62 151, which apparently lost the game.

Analysis with 703810 prefers the move as well initially, expecting 151. ... Kc3 as the answer which would lead to a perpetual check (which is what SF in fact expects: 151. Qa2+ Kc3 152. Qa3+) instead of the played 151. ... Kb3. She doesn't realize it's lost until a lot of moves later, so I think this is a position where she is completely lost in transpositions, which will only really be solved with a DAG implementation.

Naphthalin commented 4 years ago

TCEC Cup 6 Final, first game. According to SF observing the game, the endgame would have been a fortress, but Lc0 blundered with 67. ... Rf8 in 8/8/2p5/2P5/1P2R1K1/2k1P3/8/5r2 b - - 8 67. Analysis with 703810 shows Rf8 as the top move until 600k nodes, where Rf7 takes over as Rf8 is climbing in eval.

Naphthalin commented 4 years ago

CCC Lc0 - SF - SFNNUE, game 204:

[Event "More Chesse-Play: Stockfish vs Lc0 vs Stockfish+NNUE (3|2)"]
[Site "?"]
[Date "2020.07.24"]
[Round "1"]
[White "Stockfish NNUE"]
[Black "Lc0"]
[Result "1/2-1/2"]

1. b4 e5 2. Bb2 f6 3. e4 Bxb4 4. Bc4 Ne7 5. c3 Ba5 6. d4 c6 7. dxe5 fxe5 8. Bb3
Na6 9. Ba3 Ng6 10. h4 Qf6 11. Nf3 Bc7 12. Ng5 d6 13. Qh5 Bd7 14. Nxh7 Rxh7 15.
Qxh7 O-O-O 16. Qh5 Rf8 17. Qf3 Qe7 18. Qg3 Nf4 19. Nd2 b5 20. h5 Nc5 21. Kd1 Rf6
22. Kc2 a5 23. Rab1 Bb6 24. Bxc5 Bxc5 25. a4 b4 26. Qg5 Qf8 27. Bc4 d5 28. Ba6+
Kb8 29. cxb4 Bd4 30. Nf3 Ka7 31. Nxe5 Ne6 32. Nxd7 Rxf2+ 33. Kd1 Qd6 34. e5 Qxd7
35. Qg3 Kxa6 36. h6 gxh6 37. Rxh6 axb4 38. Rxb4 Bc5 39. Qd3+ Ka7 40. Rb3 Nc7 41.
Qg6 Qc8 42. Rf3 Rxf3 43. gxf3 Qa6 44. Qxc6 Qf1+ 45. Kd2 Qg2+ 46. Kd1 Qg1+ 47.
Kc2 Qf2+ 48. Kd1 Qg1+ 49. Kc2 Qf2+ 50. Kd1 Qxf3+ 51. Kc1 Qf4+ 52. Kd1 Qg4+ 53.
Kd2 Qg2+ 54. Kc1 Qf1+ 55. Kd2 Qf2+ 56. Kd1 Qd4+ 57. Kc2 Qc4+ 58. Kd1 Qb3+ 59.
Kd2 Qb4+ 60. Kd1 Qg4+ 61. Kd2 Qg2+ 62. Kc1 Qg5+ 63. Kd1 Qg1+ 64. Kc2 Qf2+ 65.
Kd1 Qd4+ 66. Kc2 Qe4+ 67. Kd1 Qd3+ 68. Ke1 Bb4+ 69. Kf2 Qd4+ 70. Kg2 Ba5 71. Qd6
Kb7 72. Rg6 Qxa4 73. Rg7 Qe4+ 74. Kh2 Qe2+ 75. Kg3 Qd3+ 76. Kh2 Kc8 77. Kg2 Qe4+
78. Kh2 Qf3 79. Qc6 Qf4+ 80. Kg2 Qe4+ 81. Kf2 Qd4+ 82. Kf3 Qd3+ 83. Kf2 Qd4+ 84.
Kf3 Qd1+ 85. Kg2 Qd2+ 86. Kf3 Qd3+ 87. Kf2 Kb8 88. Qc5 Qd2+ 89. Kg3 Kb7 90. Re7
Qd3+ 91. Kh2 Qf3 92. Qd6 Kc8 93. e6 Qf5 94. Qd7+ Kb8 95. Qd6 Kc8 96. Qd7+ Kb7
97. Rg7 Qf2+ 98. Kh3 Qf1+ 99. Kh2 Qf4+ 100. Kg2 Qe4+ 101. Kg3 Be1+ 102. Kh2 Qe2+
103. Kh3 Qf1+ 104. Kh2 Qe2+ 105. Kh3 Qe3+ 106. Kg2 Qf2+ 107. Kh3 Qf5+ 108. Kg2
Qc2+ 109. Kh3 Qc6 110. Kg4 Qc4+ 111. Kh3 Bc3 112. Rf7 Qd3+ 113. Kh4 Qe4+ 114.
Kh3 Qe3+ 115. Kh4 Qh6+ 116. Kg4 Qg6+ 117. Kf4 Bd2+ 118. Kf3 Qe4+ 119. Kf2 Be1+
120. Kg1 Ba5 121. Kh2 Qe5+ 122. Kg2 Qg5+ 123. Kh3 Qh5+ 124. Kg2 Qg4+ 125. Kh2
Qh5+ 126. Kg2 Qg4+ 127. Kh2 Qc4 128. Kh3 Qd3+ 129. Kh4 Qe4+ 130. Kh5 Qe2+ 131.
Kh6 Qc4 132. Kh7 Bc3 133. e7 Qh4+ 134. Kg6 Qg3+ 135. Kh7 Qh2+ 136. Kg6 Qg2+ 137.
Kh5 Qe2+ 138. Kh4 Qh2+ 139. Kg4 Qe2+ 140. Kg5 Qe3+ 141. Kg4 Qd4+ 142. Rf4 Qg1+
143. Kh5 Qh2+ 144. Kg6 Qxf4 145. e8=Q Qg3+ 146. Kf5 Qh3+ 147. Kg6 Qd3+ 148. Kg5
Qd2+ 149. Kg6 Qd3+ 150. Kg5 Qd2+ 151. Kg4 Qd1+ 152. Kg3 Qg1+ 153. Kh3 Qh1+ 154.
Kg3 Be1+ 155. Kg4 Qg2+ 156. Kh5 Qh1+ 157. Kg6 Qg1+ 158. Kh7 Qh1+ 159. Kg6 Qg2+
160. Kh7 Qh2+ 161. Kg6 Qc2+ 162. Kg5 Qc1+ 163. Kg4 Qd1+ 164. Kf5 Qc2+ 165. Kf4
Qf2+ 166. Kg4 Qg3+ 167. Kh5 Qf3+ 168. Kh6 Bd2+ 169. Kg7 Bc3+ 170. Kh6 Qf4+ 171.
Kh5 Qh2+ 172. Kg4 Qg2+ 173. Kh5 Qh1+ 174. Kg4 Qg2+ 175. Kh5 Qh2+ 176. Kg4 Qg1+
177. Kh3 Qh1+ 178. Kg3 Be1+ 179. Kg4 Qh4+ 180. Kf5 Qh3+ 181. Kg6 Qg2+ 182. Kh7
Qh2+ 183. Kg8 Qg2+ 184. Kf8 Bb4+ 1/2-1/2

At move 40, SF NNUE evaluates this as -4 in favor of Lc0, whose evaluation is -1.35 However, Lc0 plays the questionable 40. ... Nc7? which leads to some perpetual like position where black tries to mate (and fails) and both sides can give checks. 703810 instantly prefers the SF NNUE choice 40. ... Rxg2 41. Qf5 Rg1+ 42. Kd2 Qc7 43. Rh7 Ng7 and sticks with that, initially evaluating as -0.9, growing to -1.36 at 30M nodes. However, forcing 40. ... Nc7 even with twofold draws activated has a hard time giving drawish evals, at 24M nodes the eval is still -0.77

This will be an interesting position for a DAG implementation.

Naphthalin commented 4 years ago

CCC AlphaZero simulation match, game 174:

[Event "AlphaZero Simulation Match (10|5)"]
[Site "?"]
[Date "2020.08.07"]
[Round "1"]
[White "Stockfish"]
[Black "Lc0"]
[Result "1-0"]

1. d4 Nf6 2. c4 c5 3. d5 e6 4. Nc3 g6 5. e4 d6 6. Bd3 Bg7 7. Nf3 exd5 8. exd5
O-O 9. h3 Re8+ 10. Be3 Bh6 11. O-O Bxe3 12. fxe3 Qe7 13. e4 Nbd7 14. Qc1 Ne5 15.
Nxe5 Qxe5 16. Qf4 Kg7 17. Nb5 Rd8 18. Nxd6 Qxf4 19. Rxf4 Rxd6 20. Raf1 Bf5 21.
exf5 g5 22. R4f2 Re8 23. Re2 Rxe2 24. Bxe2 Rd7 25. g4 Re7 26. Bf3 Ne8 27. Kf2
Nd6 28. b3 Kf6 29. Rc1 a5 30. Rc3 b6 31. a4 h6 32. Rc1 Ke5 33. Re1+ Kf6 34. Rxe7
Kxe7 35. Ke3 f6 36. Kd3 Nf7 37. Be2 Kd8 38. Bd1 Kd7 39. Ke4 Kd6 40. Ke3 Kd7 41.
Bc2 Kd6 42. Kd2 Ke7 43. Kc3 Kd6 44. Kd3 Ke5 45. Ke2 Kf4 46. Kd3 Kg3 47. Ke4 Nd6+
48. Kd3 Kxh3 49. Bd1 Kg2 50. Ke3 Kf1 51. Bf3 Ke1 52. Be2 Ne8 53. Kd3 Kf2 54. Bd1
Nd6 55. Kd2 Ne4+ 56. Kd3 Nd6 57. Kd2 Kg1 58. Kd3 Kf1 59. Kc2 Kg2 60. Kd3 Kh2 61.
Bf3 Kg1 62. Bd1 Kh1 63. Bc2 Kh2 64. Ke3 Kg1 65. Bd1 Kg2 66. Bc2 Kf1 67. Bd1 Kg2
68. Bc2 Kh3 69. Bd1 Kg3 70. Bf3 Nc8 71. Be2 Kg2 72. Ke4 Kf2 73. Kd3 Nd6 74. Bd1
Ke1 75. Be2 Kf2 76. Bd1 Kg2 77. Be2 Kh1 78. Bf3+ Kh2 79. Ke3 Kg3 80. Bd1 Kh2 81.
Be2 Kg2 82. Bd3 Kh3 83. Be2 Kg2 84. Kd3 Kg3 85. Ke3 Kh4 86. Kd3 Nf7 87. Ke4 Nd6+
88. Kd3 Kg3 89. Ke3 Nf7 90. Bf3 Kh3 91. Ke2 Nd6 92. Kd3 Kg3 93. Ke3 Kh2 94. Bd1
Nf7 95. Bf3 Nd6 96. Kf2 Kh3 97. Ke1 Nf7 98. Be2 h5 99. gxh5 Kg3 100. d6 Nxd6
101. h6 g4 102. h7 Nf7 103. Bf1 Kf3 104. Be2+ Kf4 105. Bd3 Kf3 106. Be2+ Kf4
107. Bd3 Ke3 108. Bc2 Kf3 109. Kf1 Kg3 110. Be4 Kf4 111. Bd5 Nh8 112. Be6 Kf3
113. Ke1 Ke3 114. Bd7 Kf3 115. Kf1 Nf7 116. Bc8 Nh8 117. Be6 Kg3 118. Kg1 Kh3
119. Kf2 Kh2 120. Bd5 g3+ 121. Kf1 Kh3 122. Kg1 Kg4 123. Be6 Kh3 124. Kh1 Kh4
125. Kg2 Kg4 126. Bd7 Kf4 127. Bc8 Kg4 128. Be6 Kf4 129. Kh3 Kf3 130. Kh4 Kf4
131. Bd5 Kxf5 132. Kh5 Ke5 133. Kh6 Kd6 134. Kg7 Ke7 135. Kxh8 Kf8 136. Bg2 Kf7
137. Bf3 Kf8 138. Bd5 f5 139. Bb7 Kf7 140. Bg2 Kf8 141. Bd5 f4 142. Be4 Kf7 143.
Bb7 Kf8 144. Bd5 f3 145. Bxf3 Kf7 146. Bc6 Kf8 147. Bd5 g2 148. Bxg2 Kf7 149.
Bc6 b5 150. Bd5+ Kf8 151. axb5 a4 152. bxa4 Ke7 153. Kg7 Kd6 154. h8=Q Ke5 155.
Qh4 Kf5 156. a5 Ke5 157. Qf6# 1-0

Game is drawn, until Leela at move 109 moves 109. ... Kg3 in the position 8/5n1P/1p3p2/p1p2P2/P1P3p1/1P3k2/2B5/5K2 b - - 14 109, which loses the game.

Rechecking the same position with 703810 is dubious; it instantly seems to realize that 109. ... Kg3 is good for white and wants to play another shuffling move, and agrees with SF 8 on 110. Be4 as the continuation. Evals aren't decisive by any means, but +0.35 (compared to +0.06 for the other moves) and rising.

Naphthalin commented 4 years ago

TCEC S19 DivP, game 170 Lc0 - Fire https://tcec-chess.com/#div=p&game=170&season=19

1. e4 d6 2. d4 Nf6 3. Nc3 e5 4. Nf3 Nbd7 5. Bc4 h6 6. O-O c6 7. a4 Be7 8. Re1 Qc7 9. h3 O-O 10. Bf1 Re8
11. d5 cxd5 12. exd5 a6 13. Nd2 b6 14. Nc4 Rb8 15. Ne3 g6 16. Ra3 h5 17. Na2 Nc5 18. b4 Ncd7 19. Rc3 Qd8
20. Rc6 Bb7 21. b5 Nc5 22. Nc3 axb5 23. Bxb5 Kg7 24. Qf3 Bxc6 25. dxc6 Ne6 26. Ned5 Nxd5 27. Nxd5 Nc7
28. Bc4 Nxd5 29. Bxd5 f5 30. Bd2 Qc8 31. Re4 Rh8 32. Rb4 Bd8 33. g3 Bc7 34. Kg2 Ra8 35. Qb3 h4 36. g4 f4
37. f3 Qd8 38. Be1 Qe7 39. Bf2 Rhb8 40. c4 Qd8 41. Be4 Ra6 42. Qc2 Qg5 43. Qb1 Qf6 44. Be1 Rc8 45. Kh2 Qg5
46. Qb3 Re8 47. Bb1 Qf6 48. Kg1 Raa8 49. Be4 Qe6 50. Kg2 Qf6 51. Bf2 Reb8 52. Be1 Qe7 53. Qc2 Qg5 54. Qd1 Ra6
55. Rb5 Rba8 56. Bc2 Re8 57. Rd5 Raa8 58. Bb4 Rad8 59. Bc3 Ra8 60. Qe1 Qf6 61. Bb4 Re7 62. Qa1 Re6 63. Qe1 Ree8
64. Qd2 Red8 65. Qd1 Kf7 66. Bc3 Rdc8 67. Bb4 Rd8 68. Bc3 Qe7 69. Qd3 Qf6 70. Qf1 Rdc8 71. Bb4 Rf8 72. Qe1 Rfe8
73. Qd2 Rad8 74. Qd1 Kg7 75. Bc3 Ra8 76. Bb4 Red8 77. Bc3 Rac8 78. Be4 Rh8 79. Bb4 Rhd8 80. Be1 Kh6 81. Rb5 Ra8
82. Bc2 Ra6 83. Rd5 Raa8 84. Qd3 Kg7 85. Qd1 Qe7 86. Qd3 Qf6 87. Kh2 Rg8 88. g5 Qxg5 89. Qd1 Rae8 90. Be4 Qf6 
91. Rd2 Kh6 92. Qb3 Rd8 93. Rg2 Ra8 94. Bf2 Rab8 95. Qb1 Rg7 96. Rg4 g5 97. Rg1 Qe6 98. Qb3 Qf6 99. Rd1 Rgg8
100. Rd5 Rgf8 101. Rb5 Qe6 102. Bg1 Kg7 103. Bb1 Rh8 104. Bf2 Qf6 105. Be4 Qf7 106. Kg2 Rhc8 107. Kh2 Qe7
108. Qc2 Qf7 109. Rb1 Kh8 110. Bd5 Qh7 111. Qd1 Kg7 112. Be4 Qh5 113. Qc2 Rh8 114. Rb5 Qf7 115. Qb3 Rhd8
116. Bd5 Qf5 117. Be4 Qf7 118. Bd5 Qf6 119. Qc2 Rf8 120. Be4 Qf7 121. Rb4 Qe7 122. Qb2 Qd8 123. Qb3 Rf6
124. Qb1 Rf7 125. Qg1 Kf6 126. Rb2 Rg7 127. Rd2 Qg8 128. Qc1 Ke7 129. Bd5 Qe8 130. Be4 Kf6 131. Rd5 Qc8
132. Qc3 Re7 133. Rb5 Re6 134. Qb3 Re7 135. Qd1 Rg7 136. Rd5 Ke7 137. Qd2 Rg8 138. Qd3 Kf6 139. Bh7 Qe6
140. Bxg8 Qxg8 141. Kg2 Qe8 142. Qh7 Qf7 143. Qd3 Qe8 144. Rxd6+ Bxd6 145. Qxd6+ Kf5 146. c7 Ra8 147. Qd3+ Kf6
148. Bxb6 Rc8 149. Qd8+ Kf5 150. a5 Qe6 151. Qd5 Qxd5 152. cxd5 Kf6 153. d6 Ke6 154. Bc5 Kd7 155. a6 Re8
156. a7 Ra8 157. Kh2 Kc6 158. Kg2 Kd7 159. Kf2 Rg8 160. Ke2 Re8 161. Kf2 Ra8 162. Ke1 Rg8 163. Kd2 Kc6 164. Ke1 Kd7
165. Kd2 Kc6 166. Ke2 Kd7 167. Ke1 Kc6 168. Ba3 Kd7 169. Bc5 Kc6 170. Kd2 Ra8 171. Kd1 Kd7 172. Kd2 Kc6 
173. Kd1 Rg8 174. Ke2 Kd7 175. Kd3 1/2-1/2

The eval graph is a wild ride, so there supposedly were multiple positions where Lc0 missed a win. One particular position is at move 126: 1r1q4/2b2r2/1pPp1k2/4p1p1/PRP1Bp1p/5P1P/5B1K/6Q1 w - - 0 126 which was analyzed by Kovax in the Lc0 Discord and is a missed win according to SF. The winning line is

126. Qg4 Rg7 127. Qf5+ Ke7 128. a5 bxa5 129. Rb5 Qe8 130. Ba7 Rxb5 131. cxb5

where the newer J92-190 has difficulties finding 126. Qg4, 128. a5 and 130. Ba7. Movewise analysis shows (at default settings): 130. Ba7 needs about 100k nodes until it has the best eval, and is the best move from 150k (repetitions show this number to deviate between 150k and 250k) 128. a5 needs about 1.4M nodes until it has the best eval, and 2.7M nodes until it is the most visited move 126. Qg4 needs about 23M nodes until Qg4 has the highest eval, and 10M more nodes until it is the top move by N.

Given that at least the newer net finds the move but takes increasingly long to find it at earlier steps, this might be an interesting position to test progress, i.e. if future nets/search modifications find it at significantly lower nodes.

Also, this might be a strong argument for a LCB implementation; when Qg4 is discovered to be significantly better (0.65 vs 0.37) it has 3M nodes, but we still wouldn't play it until we searched for at least 10M more nodes.

Naphthalin commented 4 years ago

While not being a blunder which turns a draw into a loss, Lc0 J92-190 showed a serious oversight in game 6 of the TCEC S19 SuFi, missing the move 13. bxc5, which led to playing 12. ... Bc4 which accelerated the loss. SF's eval changed from 1.2 to 1.87, and Leela's eval also increased from 0.60 to 0.87 after being shown the move.

PGN: 1. e4 e6 2. d4 d5 3. Nc3 Nf6 4. e5 Ng8 5. a3 b6 6. Nce2 c5 7. Nf3 Ba6 8. h4 h6 9. h5 Nc6 10. c3 Qd7 11. Rb1 Nge7 12. b4 Bc4 13. bxc5 bxc5 14. dxc5 Rb8 15. Be3 Rxb1 16. Qxb1 g5 17. hxg6 Nxg6 18. Rh5 Bg7 19. Ned4 Ncxe5 20. Nxe5 Nxe5 21. Bxc4 Nxc4 22. Bf4 Qd8 23. Qb5+ Kf8 24. c6 Qb6 25. c7 Qxb5 26. Nxb5 Ke7 27. Nxa7 Bxc3+ 28. Ke2 Nb6 29. Rh3 Bf6 30. Rb3 Nc8 31. Nc6+ Kd7 32. Nb8+ Ke7 33. a4 e5 34. Bd2 Rg8 35. a5 Rg4 36. Rb7 Ra4 37. Na6 Rc4 38. Nb4 Kd6 39. Rb6+ Nxb6

Reconstructing it shows that the underlying "blind spot" is 15. Be3 which has a relatively low policy, and it took about 170k nodes in the position to find the move. Leela can find it after ~330k nodes at move 14 as well, but doesn't find it at move 13 even after millions of nodes; at 5M nodes spent on move 13, the correct 13. bxc5 only got to 72.5k nodes.

Interestingly, the MLH settings actually help here, but not enough to overcome the initial eval differences.

Finally, when testing the same position with #963, the same net can find 13. bxc5 with ease at less than 200k nodes, and give the correct continuation including 15. Be3 at 400k nodes, probably thanks to both the policy scaling and the faster eval convergence after discovering a new line. This is even fast enough to avoid playing the move 12. ... Bc4 and instead play the move 12. ... cxb4 from 700k nodes which was the move SF was expecting.

Naphthalin commented 4 years ago

TCEC S19 SuFi, game 48. Position is 2r4k/2pb1pr1/3p1n2/pNpP3p/P1P1PPpb/1P1N4/2B1R2P/6RK b - - 1 30, J92-190 plays 30. ... Rcg8 which makes SF eval jump from 0.82 to 1.71 because she misses 33. Ne1. It takes about 70k visits on Ne1 before the eval is the top move, which doesn't happen in PUCT; however, 33. Nc1 would also have been enough to dissuade Leela from playing 30. ... Rcg8` in the first place.

Naphthalin commented 4 years ago

TCEC S19 SuFi, game 52. Position is r1bq1rk1/pp2bppp/2n5/3pn1P1/3N4/2N1B1R1/PPPQ1P1P/2KR1B2 b - - 1 13; J92-190 moves instantly with 13. ... Bg4 which changes the position from already bad to lost. Reason this time is missing the move 15. Nxe6 after 14. Be2 Be6; after 500k nodes in the position, it didn't even get to 4k visits, while it takes until approx. 1M visits on 15. Nxe6 for it to be close to the top eval. This number is infeasible for PUCT.

For comparison: with #963 at defaults, it takes about 150k visits for 15. Nxe6 to get to that eval. This number still won't be reached by PUCT, but it is much more realistic.

Again in this game, at move 25 we are in the position r2q1r1k/pp2b1n1/2n1B3/6pp/3pN3/3QB2R/PPP4P/2KR4 w - - 0 25 where J92-190 fails to see 25. Ng3 until a few million nodes. When forcing it, it takes less than 20k nodes for +1.0 and around 100k nodes for +2.0 (with #963 even only <10k nodes to realize that it's completely winning for white,) but until then it is evaluated around -0.7, but it takes until 1.7M nodes for the move to have the highest eval, and then another 1.6M until it is the top move.

Naphthalin commented 4 years ago

Bad news for Leela fans at TCEC is good news for this issue. Game 54, move 17 or 18 Leela plays a move which loses the game; However, she doesn't realize it until 19. c5 is played. Analyzing the position r1bqr1k1/pp3nbp/2pp2p1/5p2/2P1PP2/2N4P/PPQ1B1P1/3RBR1K w - - 2 19 until 3M nodes, c5 only got to 93k nodes, but it takes until 500k nodes spent on the move until it has the best eval in the position (Bd3 is at +0.37 at 3M nodes), which is impossible to reach even under TCEC conditions as the eval drops from +0.32 at 93k nodes to +0.16 at 360k nodes, and only climbs slowly from there. Estimated number of visits to have c5 as top eval move: 60M nodes.

For comparison: #963 only needs 35k nodes spent on 19. c5 to have it as the top move, which takes about 1.4M nodes in this position without forcing its exploration. However, its eval drops again, and only from 160k nodes spent on c5 it remains stable on the top.

Naphthalin commented 4 years ago

Game 56 included an evaluational oversight which luckily didn't change the course of the game. At move 41, position is 7r/2qr1pbk/3p2p1/p2Bn3/3BP1P1/1PP5/4QR2/5R1K w - - 1 41. White's 41. Kg1 wins, but it takes Leela ~50k nodes spent on the move to realize that all possible answers are indeed losing for black. However when searching on the position, even at 2M nodes the winning move (only one? Someone would have to check SF) J92-190 has only spent 4.6k visits on the node, meaning that it would take until ~100M nodes to find the move on her own.

Mr-Twave commented 3 years ago

There's not really a specific point in this game that I can pick out as a blunder, however, lc0 has a very high evaluation for black, then loses its advantage slowly throughout this game. Evaluation goes slowly from -25 on move 38, to -0.67 on move 94. Lc0 ultimately wins this game only because Wasp is not equipped to handle nuance in these endgames. If Stockfish was plays as black, it easily draws this game against lc0 from around moves 80-97.

The problem in this game looks like a value problem. Lc0 knows when some positions are winning, but the value is not strong enough to distinguish which winning moves are best, especially when the 50 move rule is far away.

8/4k3/8/4B1p1/p1r3Pp/P6P/5P1K/8 b - - 0 73 Move 78 looks like a suspect move according to Stockfish when given the moves occurring in the game. Lc0 plays Kf7?! which fails because the 50 move rule doesn't allow black to win. 78... Rb7 79. Bd4 Kd5 80. Bf6 Rb3 81. Be7 Ke4 82. Bf8 Kd3 83. Bd6 Ke2 84. Be7 Rd3 85. Bb4 Rd4 86. Ba5 Kd3 87. Kf1 Kc2 88. Kg2 Kb3 89. f4 gxf4 image

[Event "Odds Ladder (3|2)"]
[Site "?"]
[Date "2020.12.13"]
[Round "1"]
[White "Wasp"]
[Black "Lc0"]
[Result "0-1"]
[ECO "A00"]
[GameDuration "00:14:27"]
[GameEndTime "2020-12-13T09:52:57.758 PST"]
[GameStartTime "2020-12-13T09:38:30.262 PST"]
[Opening "Durkin's attack"]
[PlyCount "282"]
[TimeControl "180+2"]

1. Na3 Nc6 2. Rb1 Nb4 3. Nf3 Nd5 4. Ng1 Nc3 5. Nf3 Nxb1 6. Nxb1 Nf6 7. Ng1 Ng8
8. e4 d5 9. exd5 Qxd5 10. Nc3 Qe5+ 11. Be2 Bg4 12. d4 Bxe2 13. Ngxe2 Qa5 14. O-O
Nf6 15. a3 e6 16. Re1 Be7 17. Nf4 O-O 18. b4 Qa6 19. d5 Qc4 20. Qd3 Qxd3 21.
Nxd3 Rac8 22. dxe6 c5 23. exf7+ Rxf7 24. b5 Bd8 25. Kf1 Ba5 26. Bb2 c4 27. Ne5
Re7 28. Re2 Rce8 29. Nxc4 Bxc3 30. Rxe7 Rxe7 31. Bxc3 Rc7 32. Bxf6 Rxc4 33. Be5
Rxc2 34. Bb8 a6 35. bxa6 bxa6 36. Be5 a5 37. Ke1 Kf7 38. h3 Ra2 39. Bd6 g5 40.
Bc5 h5 41. g4 h4 42. Kf1 Rc2 43. Bb6 a4 44. Bd4 Rc4 45. Bb2 Rc5 46. Kg2 Ke7 47.
Bd4 Rb5 48. Bh8 Kf7 49. Bd4 Ke8 50. Kh2 Kd7 51. Bg7 Rb8 52. Bd4 Rc8 53. Kg2 Ke8
54. Be5 Rc6 55. Bb2 Kf7 56. Bd4 Ke8 57. Be5 Kf7 58. Kf1 Ke7 59. Kg2 Ke8 60. Bd4
Rc8 61. Bf6 Rc5 62. Ba1 Kf7 63. Bd4 Rb5 64. Kh2 Ke6 65. Kg2 Ke7 66. Bg7 Rc5 67.
Bd4 Rc7 68. Kh2 Rc4 69. Bg7 Rc5 70. Ba1 Rc1 71. Bg7 Rc5 72. Bd4 Rc4 73. Be5 Rc5
74. Bd4 Rc7 75. Kg2 Ke6 76. Bb2 Ke7 77. Bd4 Ke6 78. Ba1 Kf7 79. Bd4 Ke7 80. Be3
Rc4 81. Bxg5+ Ke8 82. Bf6 Rc5 83. Bd4 Rc4 84. Bg7 Rc5 85. Bf6 Kf7 86. Bb2 Ke8
87. Bf6 Kf7 88. Ba1 Kf8 89. Bd4 Rc6 90. Kf3 Kf7 91. Kg2 Ke6 92. Bb2 Ke7 93. Bd4
Rc2 94. Kf3 Kf7 95. Kg2 Rc4 96. Be5 Ke7 97. Kf3 Ke6 98. Bb2 Rc2 99. Bd4 Rd2 100.
Ke4 Kf7 101. Bc5 Rb2 102. Kf3 Rb5 103. Bd6 Ke6 104. Bf8 Rb8 105. Bc5 Kd5 106.
Be7 Rb7 107. Bf6 Rb3+ 108. Kf4 Rxa3 109. f3 Ke6 110. g5 Ra2 111. Bd4 Rg2 112.
Ke4 a3 113. f4 Kf7 114. Ke5 a2 115. Kf5 Rg3 116. Ke4 Rxh3 117. f5 Rb3 118. g6+
Kf8 119. Bc5+ Kg7 120. Bd4+ Kh6 121. Be5 h3 122. Bf4+ Kh5 123. g7 a1=Q 124. g8=Q
Rb4+ 125. Kd5 Qa5+ 126. Kd6 Rb6+ 127. Ke7 Qa7+ 128. Kf8 Rb8+ 129. Bxb8 Qxb8+
130. Kf7 Qxg8+ 131. Kxg8 h2 132. Kf7 h1=Q 133. Ke7 Qc6 134. f6 Kg6 135. f7 Qf6+
136. Kd7 Kxf7 137. Kc7 Ke7 138. Kb7 Kd8 139. Kb8 Qg7 140. Ka8 Kc7 141. Ka7 Qa1#
0-1
Naphthalin commented 2 years ago

Similar to the T60 blunder collection, this can be archived.