search

lichess-org / lila

♞ lichess.org: the forever free, adless and open source chess server ♞
https://lichess.org
GNU Affero General Public License v3.0
15.67k stars 2.28k forks source link

integrate chessdb, a database of computer evaluations, moves and pvs #5200

Open niklasf opened 5 years ago

niklasf commented 5 years ago

examples:

http://www.chessdb.cn/cdb.php?action=queryall&board=rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR%20w%20KQkq%20-%200%201&json=1 http://www.chessdb.cn/cdb.php?action=querypv&board=rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR%20w%20KQkq%20-%200%201&json=1

github: https://github.com/noobpwnftw/chessdb

cc @noobpwnftw

noobpwnftw commented 5 years ago

There are two more endpoints that may or may not be useful: http://www.chessdb.cn/cdb.php?action=queue&board=rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR%20w%20KQkq%20-%200%201&json=1 This queues the given position(and subsequent positions) for analysis with higher priority.

http://www.chessdb.cn/cdb.php?action=queryscore&board=rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR%20w%20KQkq%20-%200%201&json=1 This gives centipawn evaluation score of the given position without other information(useful to draw an eval graph).

niklasf commented 5 years ago

Work in progress: https://github.com/ornicar/lila/compare/master...niklasf:chessdb-cn

The queryall endpoint currently responds with a lone {:

curl -v https://www.chessdb.cn/cdb.php\?action\=queryall\&boad\=rnbqkbnr%2Fpppppppp%2F8%2F8%2F8%2F8%2FPPPPPPPP%2FRNBQKBNR%20w%20KQkq%20-%200%201\&json\=1
*   Trying 170.33.8.57:443...
* TCP_NODELAY set
* Connected to www.chessdb.cn (170.33.8.57) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=chessdb.cn
*  start date: Jun 18 19:29:10 2019 GMT
*  expire date: Sep 16 19:29:10 2019 GMT
*  subjectAltName: host "www.chessdb.cn" matched cert's "www.chessdb.cn"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55854eee3f30)
> GET /cdb.php?action=queryall&boad=rnbqkbnr%2Fpppppppp%2F8%2F8%2F8%2F8%2FPPPPPPPP%2FRNBQKBNR%20w%20KQkq%20-%200%201&json=1 HTTP/2
> Host: www.chessdb.cn
> User-Agent: curl/7.65.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Sat, 22 Jun 2019 20:00:53 GMT
< content-type: text/html; charset=UTF-8
< content-length: 1
< x-frame-options: SAMEORIGIN
< cache-control: no-cache
< pragma: no-cache
< access-control-allow-origin: *
< x-cache: MISS
< 
* Connection #0 to host www.chessdb.cn left intact
{%
noobpwnftw commented 5 years ago

It seems a typo: boad -> board.

niklasf commented 5 years ago

Oh, my bad. Perhaps the server should respond 400?

noobpwnftw commented 5 years ago

I'm working on it, it should output a proper json at least. EDIT: Done.

niklasf commented 5 years ago

Thanks!

Next issue: The Content-Type header must be application/json in order for XHR to work properly. It is currently content-type: text/html; charset=UTF-8.

noobpwnftw commented 5 years ago

Fixed.

noobpwnftw commented 5 years ago

A few notes on the data: score field has a range of +-30000, where values above 10000 or below -10000 represents mate score in plies towards their corresponding max value. So a mate in 4 plies for the winning side has a score of 29996, a mate in 4 plies for the losing side has a score of -29996, otherwise they are centipawn evaluations of the playing side. rank field has three values: 0,1,2. 0 means it's a bad move, 1 means it's a good move, 2 means it's among the best moves. notes field has the following format: <symbol of rank> (<num1>-<num2>), where the symbol of rank translates to !=2, *=1, ?=0 according to rank field, num1 is the number of explored(known) moves of the position after making the corresponding move at the current position, num2 is the number of rank=2 moves plus the number of rank=1 moves among explored moves. winrate field is the conversion of centipawn score with this formula: 100 / ( 1 + exp( -centipawn / 330 ) ), a mate score does not have this field.

The decision of each moves' rank is based on an aspiration window scaled by the highest centipawn evaluation of explored moves and the centipawn evaluation is back-propagated from subsequent positions by taking weighted average of scores among their explored moves sieved by a similar aspiration window, which also considers the number of moves being sampled.

niklasf commented 5 years ago

Thanks for the explanation.

Meanwhile I am done with most of the plumbing. Deployed to the test server (https://lichess.dev/analysis#explorer, select chessdb.cn after clicking the little gear icon).

Of course that's just the raw data. Still need to present it more nicely and clean up the patch.

Changes: https://github.com/ornicar/lila/compare/master...niklasf:chessdb-cn

noobpwnftw commented 5 years ago

There may also a case of "stalemate" in the status field returned.

noobpwnftw commented 5 years ago

I'm not quite familiar with GUI coding, here are my thoughts: winrate is an optional field, it should be declared as winrate?: string;. Rendering could have be limited to moves that have rank > 0 to filter out junk ones. Move list could be displayed with two columns: Move and Advantage, where Move shows their SAN notation, Advantage is a horizontal percentage bar that shows winning chances(needs conversion from POV of the playing side to white & black), moves without winrate field have 100% chances of winning/losing.

ornicar commented 5 years ago

Really cool API but I'm not sure how to make it useful within lichess analysis tools for now

noobpwnftw commented 5 years ago

One quick use case is as a data source to opening explorer, which is partially done so far. Another is to populate the eval cache, which can in turn provide more analysis results aside fishnet.

vondele commented 1 year ago

so, in the last 3 years, the number of positions in the database has grown to 17,308,176,712 positions, I think this among the most extensive 'opening' databases (often up to 50plies deep). I believe it would be valuable to consider the integration once more.

dav1312 commented 1 year ago

The database was cleared in January and now uses only the new evaluation (before it was a mix of new and old). 10 billion positions is around 150GB. Even just using a snapshot of the database instead of a live one could bring big improvements to the cloud analysis.

The current cloud uses Stockfish "14+" and the way it works causes a few issues that can be seen in this game:

This can easily confuse the final user, that sees that they played all of the best moves but the evaluation keeps going down and the accuracy is much lower than what it should be.

ChessDB has:

Lichess ChessDB
Startpos 0.15 0.00
1. e4 0.4 0.00
... e5 0.4 0.01
2. Nf3 0.3 0.00
... Nf6 0.5 0.09
3. Nxe5 0.3 0.07
... Nc6 1.4 1.47
4. Nxc6 2.3 1.47
... dxc6 2.5 1.48
5. d3 1.7 1.48
... Bd5 2.6 1.62
6. Be2 1.5 1.62
... h5 2.7 2.13
7. c3 3.0 2.13
... Bxf2 7.0 4.21
8. Kxf2 4.3 4.21
... Ng4+ 4.9 4.78
9. Kg1 5.0 4.78
... Ne3 6.7 6.29
10. Bxe3 6.1 6.29
White
Accuracy		 84% 100%

chart

avannaa commented 1 year ago

any current thoughts on this? i think it's a bit of a big deal :o

as a master, i've relied on this for a lot of my own analysis in the past few months, and it sounds like it could save a fair amount of server resources. perhaps the most confusing part is that it's too deep~ like, it's probably funny to look at the initial position and have the eval be like "yeah 0.00", and how it stays that way even after e.g. 1 e4 c5 2 h4?! (notation mine), would probably be a bit of a culture shock to many people. also, i suppose that the "eval discontinuity" when you leave chessdb could be funny in many cases, but that already happens in arguably worse ways with cloud..

dav1312 commented 1 year ago

also, i suppose that the "eval discontinuity" when you leave chessdb could be funny in many cases, but that already happens in arguably worse ways with cloud..

Yes Lichess already has eval discontinuity issues with their server analysis since they mix cloud (sf 14+) evals with new stockfish evals. If chessdb was used the issue would change, the discontinuity would happen with local analysis but not with server analysis. They could limit the use of chessdb just for server analysis and keep the existing cloud for local analysis.

If chessdb is used for local analysis the interface could be as simple as something like the engine manager

image

Chessdb now has over 22 billion positions, which should be around 330GB of data

Siderite commented 1 year ago

I am seeing a lot of eval differences between SF14+ and SF16 and the setup of the engines for chessDb or any other source might be different, too.

I believe that instead of "CLOUD" the source of the cloud eval should be shown, like "SF16" or "SF14+", perhaps with the db source in the title. Maybe the user could choose the sources they want to use from a list of implementations of a common interface?

dav1312 commented 1 year ago

I am seeing a lot of eval differences between SF14+ and SF16 and the setup of the engines for chessDb or any other source might be different, too.

I explained this in my first comment.

Cloud evals are SF14+ evals so they shouldn't be compared or mixed together with the new SF15.1/16 evals like they do for server analysis. Chessdb only uses normalized evals so the evaluations it has stored will be comparable with newer versions of Stockfish.

The Lichess cloud evals are independent of each other and they can have widely different evaluations because the analysis depth of each position is totally different. If anything, they might be wasting a lot of space storing the evaluation and pv of each position when they only need leaf evals and can create pvs on the fly.

Siderite commented 1 year ago

Could/would you allow the www.chessdb.cn domain in the CSP (Content-Security-Policy meta tag) of lichess so that we can test various usages of this?

Siderite commented 1 year ago

Example Explorer with cp values from chessdb: image

dav1312 commented 3 months ago

Chessdb now has over 22 billion positions, which should be around 330GB of data

Update: chessdb now has 43 billion positions, 645GB of data

Siderite commented 3 months ago

Oh, hi, Dav! Wanted to talk to you for a long time! Great work! I use your project in LiChess Tools to great effect!

vondele commented 3 months ago

Time for some update here. In the past few days, we have have been working on proof of concept code to access an off-line copy of cdb. This turns out to be easier than expected. The PoC code is right now at https://github.com/vondele/cdbdirect An new offline copy of cdb has been made available as well (courtesy of @noobpwnftw) and can be downloaded in roughly 24h time (about 800GB), see repo above for instructions.

On a desktop PC (M.2 SSD, some RAM), one can query about 0.5M positions per second, getting for each position the top moves all scored. The repo has example code that shows how to do that, with a very simple API.

To illustrate what can be done, the following is an example (favorable) output

Loading: lichess_db_standard_rated_2024-03_plies50Early4Count10.epd
Probing 10464957 fens with 32 threads.
known fens:   10462332
unknown fens: 2625
scored moves: 137555533
Required probing time:         20.6835 sec.
Required time per fen: 1.97645 microsec.

Showing that 99.97% of the 'popular positions' (seen at least 10 times in March 2024), within the first 50 plies are in the db scored. Obviously that hit rate goes down as positions become less popular.

I'm pretty sure this could be used for some nice applications like near instantly score games for as long as they are in the DB, or find the first blunder or so.

More discussion on this topic in the SF discord, around https://discord.com/channels/435943710472011776/1101022188313772083/1273910582894006284

niklasf commented 2 months ago

For reference: https://github.com/niklasf/lila-cloudeval (experimenting with more ways to query the database dump)