search

CuriosAI / sai

SAI: a fork of Leela Zero with variable komi.
GNU General Public License v3.0
104 stars 11 forks source link

suggest begin to train the 20*192 size net #124

Open l1t1 opened 4 years ago

l1t1 commented 4 years ago

since 20224 20256 are all tested by others, and 12*128 stalled

Vandertic commented 4 years ago

Thank you for asking. Current structure is in fact 12x256, not 12x128. We are going to train 20x256 next, but we are having some problems we want to solve before, just to be sure that we are not stalling because of them.

l1t1 commented 4 years ago

when will you swich to 20x256 net? stronger net will get more contributors

Vandertic commented 4 years ago

Thank you as always for you interest, @l1t1. We are trying to train the 20x256 network, but there are technical difficulties which are slowing down the procedure.

First of all, "RAM" minibatch had to be reduced from 128 to 64 positions, because otherwise the training doesn't fit in the GPU memory (which, by the way is a RTX 2080ti with 11Gb). This in turn changed something in the training, and the learning wouldn't even start with the same hyperparameters we used for the smaller structures. We found that by decreasing the learning rate this could be corrected, but this in turn gave overfitting after some generations.

So it's a tricky exercise, but we are spending all time on it at the moment. I hope we will be able to give good news in a few days and have a 20x256 network ready in three weeks.

l1t1 commented 4 years ago

rtx 3090 has 24GB 20200828_203608_995

Vandertic commented 4 years ago

Awesome. It would be great if we could find the money to buy one, but I don't see it coming for now...