ZeroAgent becomes weaker after learning

[y-kkky]

Steps I've done to start training ZeroAgent for 9x9 board size:

At first, I've put my model in a separate file:

Model

def zero_model(board_size): encoder = zero.ZeroEncoder(board_size) board_input = Input(shape=encoder.shape(), name='board_input') pb = board_input for i in range(4): pb = Conv2D(64, (3, 3), padding='same', data_format='channels_first')(pb) pb = BatchNormalization(axis=1)(pb) pb = Activation('relu')(pb) policy_conv = Conv2D(2, (1, 1), data_format='channels_first')(pb) policy_batch = BatchNormalization(axis=1)(policy_conv) policy_relu = Activation('relu')(policy_batch) policy_flat = Flatten()(policy_relu) policy_output = Dense(encoder.num_moves(), activation='softmax')( policy_flat) value_conv = Conv2D(1, (1, 1), data_format='channels_first')(pb) value_batch = BatchNormalization(axis=1)(value_conv) value_relu = Activation('relu')(value_batch) value_flat = Flatten()(value_relu) value_hidden = Dense(256, activation='relu')(value_flat) value_output = Dense(1, activation='tanh')(value_hidden) model = Model( inputs=[board_input], outputs=[policy_output, value_output]) return model

Then, I initialized my ZeroAgent this way:

encoder = zero.ZeroEncoder(9)
model = zero_model(9)
agent = zero.ZeroAgent(model, encoder, rounds_per_move=50, c=2.0)
with h5py.File('original_zero.h5', 'w') as outf:
    agent.serialize(outf)

I created play_train_eval_zero.py by the example of other play_train_eval.py s: https://pastebin.com/HUHnYWBX

Example configuration: --agent original_zero.h5 --num-workers 6 --games-per-batch 500 --board-size 9 --games-eval 60

And what I see is the degradation of my bot during learning:

Reference: original_zero.h5 Learning_agent: original_zero.h5 Total games so far 0 Won 33 / 60 games (0.550) New reference is agent_00000500.hdf5 Reference: agent_00000500.hdf5 Learning_agent: agent_cur.hdf5 Total games so far 500 Won 25 / 60 games (0.417) Reference: agent_00000500.hdf5 Learning_agent: agent_cur.hdf5 Total games so far 1000 Won 10 / 60 games (0.167) Reference: agent_00000500.hdf5 Learning_agent: agent_cur.hdf5 Total games so far 1500 Won 11 / 60 games (0.183)

I tried different configurations and it is a common pattern for all of them and I have no idea.

I had some assumptions:

1. That somehow I copy the wrong file after training and my reference becomes stronger while learning_agent remains the same. It is not true - I added an additional step to save files after each round if learning_agent lost evaluation, and checked hashes of model files - they were the same for reference_agent (as it should be), so it is not the case.
2. I have also checked manually that encoder works fine, also that experience collector works fine, looks like they are ok. UPD. I found out that I'm using komi 7.5 for 9x9 board, but maybe this is ok

The last assumption is that I do very small amount of simulations, but I'm afraid to run this on paid GPU server, to check this assumption - cause it is very strange pattern that bot becomes weaker with each round.

[y-kkky]

What parameter should I tune for real training? As I understand there are 4 parameters to play with:

1. model
2. encoder
3. games per batch
4. rounds per move

How to decide which should I sacrifice?

[computer-idol]

Do you train AlphaGo that is described in chapter 13? If completed，can you share the model files?

[y-kkky]

@computer-idol AlphagoZero from chapter 14