search

lichess-org / lila-openingexplorer

Opening explorer for lichess.org that can handle all the variants and trillions of unique positions
http://lichess.org/analysis#explorer
GNU Affero General Public License v3.0
135 stars 34 forks source link

tune defrag parameter #25

Closed niklasf closed 8 years ago

niklasf commented 8 years ago

Kyoto can be tuned to execute one defragmentation operation per n writing operations. And it can be totally worth it (see below for a full out-of-band defrag). We should play with the settings.

Before:

ubuntu@lichess-cloud:~/tonyro_successful_import$ du -h *
312M    LiChess 2200 DB.7z?dl=0
1.7G    LiChess 2200 DB.pgn
3.5G    master.kct
1.3G    master-pgn.kct

Full defrag:

ubuntu@lichess-cloud:~/tonyro_successful_import$ kctreemgr defrag master-pgn.kct 
kctreemgr: [WARN]: ./kchashdb.h: 2717: reorganize_file: -: reorganizing the database
kctreemgr: [WARN]: ./kcplantdb.h: 3642: reorganize_file: master-pgn.kct: reorganizing the database
ubuntu@lichess-cloud:~/tonyro_successful_import$ kctreemgr defrag master.kct 
kctreemgr: [WARN]: ./kchashdb.h: 2717: reorganize_file: -: reorganizing the database
kctreemgr: [WARN]: ./kcplantdb.h: 3642: reorganize_file: master.kct: reorganizing the database

After:

ubuntu@lichess-cloud:~/tonyro_successful_import$ du -h *
312M    LiChess 2200 DB.7z?dl=0
1.7G    LiChess 2200 DB.pgn
1.5G    master.kct
991M    master-pgn.kct

Stats:

Pack format 0: 46125923 nodes 
Pack format 1: 2609441 nodes 
Pack format 2: 520405 nodes 
Pack format 3: 6800 nodes 
Pack format 4: 2 nodes 
Pack format 5: 0 nodes 
Pack format 6: 0 nodes 
Unique positions: 49262571
niklasf commented 8 years ago

Should be fine around MAX_PLIES, so about 1 defrag operation per game. Almost no slow down on my machine.