[Enhancement] Training Data Tools

[Quinniboi10]

A few simple python scripts that make it easy for users to download training data from the Leela Chess Zero website.

[mooskagh]

Thanks for your PR. However, I'm not quite convinced about merging it at this point. It appears that each script might be simplified into a single shell command, like turning cleanlinks into [cut -f 1 -d ' ' linksdirty.txt > links.txt].

Also, I'm not sure if adding an arbitrary file list to the repo is necessary. So, I'm a bit uncertain about the overall usefulness of this tool.

[Quinniboi10]

OK. Thanks. I'm not great at scripts outside of python, so I should definitely consider learning bash. Thank you for your time. I would like to add my latest Lc0 project, a simple script that gives the user a progress bar of the training. "selfplay.sh" is the script for training.

import os
from multiprocessing import *
import subprocess
import time
from tqdm import tqdm

def cmd(command, exit):
    os.system(command)

def report_sp(exit):
    time.sleep(10)
    bar = tqdm(total=15000, desc='Selfplay Games')
    while True:
        output = int(subprocess.check_output("find ~/.cache/lc0/ -type f -print | wc -l", shell=True).decode("utf-8"))
        bar.update(output - bar.n)
        if output >= 15000:
            exit.value = True
            print("Exit")
        time.sleep(0.1)

if __name__ == '__main__':
    exit = Value('b', False)
    selfplay = Process(target=cmd, args=("bash selfplay.sh >/dev/null 2>&1", exit))
    report = Process(target=report_sp, args=(exit,))
    selfplay.start()
    report.start()
    try:
        while True:
            if exit.value:
                print("Stopping selfplay.")
                break
            time.sleep(0.1)
    except:
        pass

    finally:
        selfplay.terminate()
        report.terminate()
        os.system("pkill lc0")

selfplay.sh:

../../lc0/build/release/lc0 selfplay --training=true --visits=10000 --cpuct=1.32 --cpuct-at-root=1.9 --root-has-own-cpuct-params=true --resign-percentage=2.0 --resign-playthrough=20 --temperature=1.5 --temp-endgame=0.60 --tempdecay-moves=60 --tempdecay-delay-moves=20 --temp-cutoff-move=40 --temp-visit-offset=-0.8 --fpu-strategy=reduction --fpu-value=0.26 --fpu-strategy-at-root=absolute --fpu-value-at-root=1.0 --minimum-kldgain-per-node=0.000050 --policy-softmax-temp=1.45 --resign-wdlstyle=true --noise-epsilon=0.1 --noise-alpha=0.12 --sticky-endgames=true --moves-left-max-effect=0.2 --moves-left-threshold=0.0 --moves-left-slope=0.007 --moves-left-quadratic-factor=0.85 --moves-left-scaled-factor=0.15 --moves-left-constant-factor=0.0 --task-workers=0 --games=16000```