Train results on chapter 7

[ghost]

Hello, I'm going through your book to understand and implement from scratch the whole repository. I'm really enjoying this lecture. I just finished chapter 7 training a bot using the CNN. I used the OnePlaneEncoder with 100 games from training set and succesfully reached about 95% accuracy on validation set after 200 epochs (optimizer=sgd, loss=categorical_crossentropy, activation=softmax).

I wanted to try with 10000 samples in training set to better generalize the model, using the same configuration, but I have 70% accuracy on training set and 40% on validation set. I didin't think it would overfit with more data samples.

Do you have some configuration parameters for obtain good accuracy results on a larger dataset (like mine of 10000 games)? Thanks in advance and again my compliments for your great work!

[maxpumperla]

@Zaxorn Thank you, much appreciated. Yeah, Go is massively complex and 10k samples are basically nothing. :D

@macfergus do you have anything documented for experimental setup? I sure didn't and regret it, but Kevin ran a few more extensive experiments in the end, maybe he can provide you with an advanced starting point.

Best, Max

[macfergus]

Hi @Zaxorn !

One important thing we did not note in the book is to clear out the .npy files in between runs. Otherwise, data from the previous run can get mixed in the with the current run, and it is even possible to leak data from the training set to the validation set. So when you see unlikely results like 70% accuracy with a small network and data set, I suspect that's what is happening. Sorry about this omission. Just do rm data/*.npy after each run. (You don't need to delete the .tar and .tar.xz files)

I got reasonable results with these settings:

• num_games = 15000
• 100 test set games, change the line to test_generator = processor.load_go_data('test', 100, use_generator=True) -- this is just to speed up the validation phase, and won't affect the training
• switch the optimizer from sgd to adagrad (cheating a little bit, we cover adagrad in a later chapter -- this just lets you get the same result in fewer epochs)
• 100 epochs
• clear out any npy files before the run

These gave me 20% accuracy on the training set of 3 million positions, and similar on the validation set.

At this point it's not a super strong bot, but it mostly plays things that look like real go moves:

Of course, higher num_games will just be better, i.e. 50 epochs at num_games = 30000 will be better than 100 epochs at num_games = 15000. (However, I would try a larger network and a multi-plane encoder before trying a really huge training set)

[ghost]

Hello, and thanks for both the replies. Yes, I was aware of cleaning the .npy files between runs (I splitted the data generation and training phase in my implementation, if I want to retrain with a specific dataset for example).

Regarding the settings thank you very much, I can obtain the same results with the oneplane encoder. I'll try next a very big dataset (100k samples probably) with a sevenplane encoder and a large network to obtain (after days of training probably) a better starting point for the reinforcement learning part of the book (just started chapter 9).

Thanks again

[hyu2000]

Hi Kevin & Max, I saw code/agents/betago.hdf5 in the repo which seems quite good. Is this model trained on 15k games you mentioned above? Thanks

[arisliang]

I'm a bit confused on the expected performance for the small network. How is it that training on 100 games can give 95% accuracy on validation set, yet for 15k games, the performance is only 20%?

[nctran108]

I tried the 100 games and only get 3% accuracy.

[jaxvanyang]

I tried the 100 games and only get 3% accuracy.

Same here, this confused me for days. It seems like a printing error, you should read this #108.

tl;dr In train_generator.py, set num_games = 1000 and num_epochs = 50. This should get you training set accuracy over 80%.