search

maxpumperla / deep_learning_and_the_game_of_go

Code and other material for the book "Deep Learning and the Game of Go"
https://www.manning.com/books/deep-learning-and-the-game-of-go
987 stars 390 forks source link

[Chapter 7] train_generator problems #66

Closed y-kkky closed 4 years ago

y-kkky commented 4 years ago

I have problems with training of the small model from chapter 7 (on 100 games sample), and also with already precomputed weights (code/checkpoints/small_model_epoch_5.h5). Model with 5 epochs is training too fast (~1 minute for epoch) both for train-generator.py and for train-consolidate.py (with and without generator accordingly)

train-generator.py from dlgo.data.parallel_processor import GoDataProcessor from dlgo.encoders.oneplane import OnePlaneEncoder from dlgo.networks import small from keras.models import Sequential from keras.layers.core import Dense go_board_rows, go_board_cols = 19, 19 num_classes = go_board_rows * go_board_cols num_games = 100 encoder = OnePlaneEncoder((go_board_rows, go_board_cols)) # <1> processor = GoDataProcessor(encoder=encoder.name()) # <2> generator = processor.load_go_data('train', num_games, use_generator=True) # <3> test_generator = processor.load_go_data('test', num_games, use_generator=True) input_shape = (encoder.num_planes, go_board_rows, go_board_cols) network_layers = small.layers(input_shape) model = Sequential() for layer in network_layers: model.add(layer) model.add(Dense(num_classes, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy']) epochs = 500 batch_size = 128 model.fit_generator(generator=generator.generate(batch_size, num_classes), epochs=epochs, steps_per_epoch=generator.get_num_samples() / batch_size, validation_data=test_generator.generate(batch_size, num_classes), validation_steps=test_generator.get_num_samples() / batch_size) #,callbacks=[ModelCheckpoint('../checkpoints/small_model_epoch_{epoch}.h5')]) # <5> model.evaluate_generator(generator=test_generator.generate(batch_size, num_classes), steps=test_generator.get_num_samples() / batch_size)
train-consolidate.py from dlgo.data.parallel_processor import GoDataProcessor from dlgo.encoders.oneplane import OnePlaneEncoder from dlgo.networks import small from keras.models import Sequential from keras.layers.core import Dense go_board_rows, go_board_cols = 19, 19 num_classes = go_board_rows * go_board_cols num_games = 100 encoder = OnePlaneEncoder((go_board_rows, go_board_cols)) # <1> processor = GoDataProcessor(encoder=encoder.name()) # <2> X_train, Y_train = processor.load_go_data('train', num_games, use_generator=False) X_test, Y_test = processor.load_go_data('test', num_games, use_generator=False) input_shape = (encoder.num_planes, go_board_rows, go_board_cols) network_layers = small.layers(input_shape) model = Sequential() for layer in network_layers: model.add(layer) model.add(Dense(num_classes, activation='softmax')) model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy']) epochs = 5 batch_size = 128 model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(X_test, Y_test)) score = model.evaluate(X_test, Y_test, verbose=0) print(f"score1: {score}")

Model learning too fast comparing to 14053s per epoch from book (I am training on CPU tf) + looks like it is not learning at all.

Epoch 1, 2 ... 18 Epoch 1/500 1/88 [..............................] - ETA: 1:10 - loss: 5.8850 - acc: 0.0000e+00 2/88 [..............................] - ETA: 52s - loss: 5.8871 - acc: 0.0000e+00 3/88 [>.............................] - ETA: 45s - loss: 5.8893 - acc: 0.0000e+00 4/88 [>.............................] - ETA: 42s - loss: 5.8888 - acc: 0.0000e+00 5/88 [>.............................] - ETA: 39s - loss: 5.8893 - acc: 0.0000e+00 6/88 [=>............................] - ETA: 38s - loss: 5.8891 - acc: 0.0013 7/88 [=>............................] - ETA: 36s - loss: 5.8891 - acc: 0.0011 8/88 [=>............................] - ETA: 35s - loss: 5.8895 - acc: 9.7656e-04 9/88 [==>...........................] - ETA: 34s - loss: 5.8892 - acc: 8.6806e-04 10/88 [==>...........................] - ETA: 34s - loss: 5.8890 - acc: 7.8125e-04 11/88 [==>...........................] - ETA: 33s - loss: 5.8891 - acc: 0.0014 12/88 [===>..........................] - ETA: 33s - loss: 5.8893 - acc: 0.0013 13/88 [===>..........................] - ETA: 32s - loss: 5.8894 - acc: 0.0012 14/88 [===>..........................] - ETA: 31s - loss: 5.8894 - acc: 0.0017 15/88 [====>.........................] - ETA: 31s - loss: 5.8892 - acc: 0.0026 16/88 [====>.........................] - ETA: 30s - loss: 5.8893 - acc: 0.0024 17/88 [====>.........................] - ETA: 30s - loss: 5.8891 - acc: 0.0023 18/88 [=====>........................] - ETA: 29s - loss: 5.8891 - acc: 0.0022 19/88 [=====>........................] - ETA: 29s - loss: 5.8891 - acc: 0.0021 20/88 [=====>........................] - ETA: 28s - loss: 5.8892 - acc: 0.0020 21/88 [======>.......................] - ETA: 28s - loss: 5.8890 - acc: 0.0019 22/88 [======>.......................] - ETA: 27s - loss: 5.8890 - acc: 0.0021 23/88 [======>.......................] - ETA: 27s - loss: 5.8890 - acc: 0.0020 24/88 [=======>......................] - ETA: 27s - loss: 5.8888 - acc: 0.0020 25/88 [=======>......................] - ETA: 26s - loss: 5.8888 - acc: 0.0022 26/88 [=======>......................] - ETA: 26s - loss: 5.8887 - acc: 0.0021 27/88 [========>.....................] - ETA: 25s - loss: 5.8887 - acc: 0.0020 28/88 [========>.....................] - ETA: 25s - loss: 5.8889 - acc: 0.0020 29/88 [========>.....................] - ETA: 24s - loss: 5.8890 - acc: 0.0019 30/88 [=========>....................] - ETA: 24s - loss: 5.8888 - acc: 0.0021 31/88 [=========>....................] - ETA: 24s - loss: 5.8888 - acc: 0.0025 32/88 [=========>....................] - ETA: 23s - loss: 5.8886 - acc: 0.0024 33/88 [==========>...................] - ETA: 23s - loss: 5.8886 - acc: 0.0024 34/88 [==========>...................] - ETA: 22s - loss: 5.8886 - acc: 0.0023 35/88 [==========>...................] - ETA: 22s - loss: 5.8887 - acc: 0.0022 36/88 [===========>..................] - ETA: 21s - loss: 5.8887 - acc: 0.0022 37/88 [===========>..................] - ETA: 21s - loss: 5.8887 - acc: 0.0021 38/88 [===========>..................] - ETA: 21s - loss: 5.8886 - acc: 0.0021 39/88 [============>.................] - ETA: 20s - loss: 5.8887 - acc: 0.0020 40/88 [============>.................] - ETA: 20s - loss: 5.8887 - acc: 0.0020 41/88 [============>.................] - ETA: 19s - loss: 5.8887 - acc: 0.0021 42/88 [=============>................] - ETA: 19s - loss: 5.8887 - acc: 0.0020 43/88 [=============>................] - ETA: 18s - loss: 5.8887 - acc: 0.0020 44/88 [==============>...............] - ETA: 18s - loss: 5.8887 - acc: 0.0020 45/88 [==============>...............] - ETA: 18s - loss: 5.8887 - acc: 0.0019 46/88 [==============>...............] - ETA: 17s - loss: 5.8888 - acc: 0.0019 47/88 [===============>..............] - ETA: 17s - loss: 5.8888 - acc: 0.0022 48/88 [===============>..............] - ETA: 16s - loss: 5.8887 - acc: 0.0021 49/88 [===============>..............] - ETA: 16s - loss: 5.8887 - acc: 0.0021 50/88 [================>.............] - ETA: 16s - loss: 5.8888 - acc: 0.0020 51/88 [================>.............] - ETA: 15s - loss: 5.8888 - acc: 0.0020 52/88 [================>.............] - ETA: 15s - loss: 5.8888 - acc: 0.0021 53/88 [=================>............] - ETA: 14s - loss: 5.8888 - acc: 0.0022 54/88 [=================>............] - ETA: 14s - loss: 5.8887 - acc: 0.0022 55/88 [=================>............] - ETA: 13s - loss: 5.8888 - acc: 0.0023 56/88 [==================>...........] - ETA: 13s - loss: 5.8888 - acc: 0.0022 57/88 [==================>...........] - ETA: 13s - loss: 5.8888 - acc: 0.0023 58/88 [==================>...........] - ETA: 12s - loss: 5.8887 - acc: 0.0023 59/88 [===================>..........] - ETA: 12s - loss: 5.8887 - acc: 0.0024 60/88 [===================>..........] - ETA: 11s - loss: 5.8887 - acc: 0.0023 61/88 [===================>..........] - ETA: 11s - loss: 5.8886 - acc: 0.0024 62/88 [====================>.........] - ETA: 10s - loss: 5.8887 - acc: 0.0024 63/88 [====================>.........] - ETA: 10s - loss: 5.8888 - acc: 0.0024 64/88 [====================>.........] - ETA: 10s - loss: 5.8888 - acc: 0.0023 65/88 [=====================>........] - ETA: 9s - loss: 5.8887 - acc: 0.0023 66/88 [=====================>........] - ETA: 9s - loss: 5.8887 - acc: 0.0022 67/88 [=====================>........] - ETA: 8s - loss: 5.8888 - acc: 0.0023 68/88 [======================>.......] - ETA: 8s - loss: 5.8887 - acc: 0.0023 69/88 [======================>.......] - ETA: 7s - loss: 5.8887 - acc: 0.0024 70/88 [======================>.......] - ETA: 7s - loss: 5.8888 - acc: 0.0023 71/88 [=======================>......] - ETA: 7s - loss: 5.8888 - acc: 0.0024 72/88 [=======================>......] - ETA: 6s - loss: 5.8888 - acc: 0.0025 73/88 [=======================>......] - ETA: 6s - loss: 5.8888 - acc: 0.0025 74/88 [========================>.....] - ETA: 5s - loss: 5.8888 - acc: 0.0024 75/88 [========================>.....] - ETA: 5s - loss: 5.8889 - acc: 0.0024 76/88 [========================>.....] - ETA: 5s - loss: 5.8890 - acc: 0.0024 77/88 [=========================>....] - ETA: 4s - loss: 5.8890 - acc: 0.0023 78/88 [=========================>....] - ETA: 4s - loss: 5.8890 - acc: 0.0023 79/88 [=========================>....] - ETA: 3s - loss: 5.8890 - acc: 0.0023 80/88 [==========================>...] - ETA: 3s - loss: 5.8890 - acc: 0.0022 81/88 [==========================>...] - ETA: 2s - loss: 5.8890 - acc: 0.0022 82/88 [==========================>...] - ETA: 2s - loss: 5.8890 - acc: 0.0023 83/88 [===========================>..] - ETA: 2s - loss: 5.8890 - acc: 0.0023 84/88 [===========================>..] - ETA: 1s - loss: 5.8890 - acc: 0.0023 85/88 [===========================>..] - ETA: 1s - loss: 5.8890 - acc: 0.0023 86/88 [============================>.] - ETA: 0s - loss: 5.8889 - acc: 0.0023 87/88 [============================>.] - ETA: 0s - loss: 5.8889 - acc: 0.0024 88/88 [==============================] - 45s 513ms/step - loss: 5.8890 - acc: 0.0024 - val_loss: 5.8878 - val_acc: 0.0026 Epoch 2/500 1/88 [..............................] - ETA: 50s - loss: 5.8839 - acc: 0.0000e+00 2/88 [..............................] - ETA: 46s - loss: 5.8860 - acc: 0.0000e+00 3/88 [>.............................] - ETA: 42s - loss: 5.8882 - acc: 0.0000e+00 4/88 [>.............................] - ETA: 40s - loss: 5.8875 - acc: 0.0000e+00 5/88 [>.............................] - ETA: 39s - loss: 5.8880 - acc: 0.0000e+00 6/88 [=>............................] - ETA: 38s - loss: 5.8877 - acc: 0.0013 7/88 [=>............................] - ETA: 37s - loss: 5.8877 - acc: 0.0011 8/88 [=>............................] - ETA: 36s - loss: 5.8880 - acc: 0.0020 9/88 [==>...........................] - ETA: 35s - loss: 5.8877 - acc: 0.0017 10/88 [==>...........................] - ETA: 34s - loss: 5.8875 - acc: 0.0023 11/88 [==>...........................] - ETA: 34s - loss: 5.8876 - acc: 0.0021 12/88 [===>..........................] - ETA: 33s - loss: 5.8877 - acc: 0.0020 13/88 [===>..........................] - ETA: 33s - loss: 5.8877 - acc: 0.0018 14/88 [===>..........................] - ETA: 33s - loss: 5.8877 - acc: 0.0022 15/88 [====>.........................] - ETA: 33s - loss: 5.8876 - acc: 0.0021 16/88 [====>.........................] - ETA: 33s - loss: 5.8878 - acc: 0.0020 17/88 [====>.........................] - ETA: 32s - loss: 5.8876 - acc: 0.0018 18/88 [=====>........................] - ETA: 32s - loss: 5.8875 - acc: 0.0017 19/88 [=====>........................] - ETA: 32s - loss: 5.8875 - acc: 0.0016 20/88 [=====>........................] - ETA: 31s - loss: 5.8876 - acc: 0.0016 21/88 [======>.......................] - ETA: 30s - loss: 5.8875 - acc: 0.0015 22/88 [======>.......................] - ETA: 30s - loss: 5.8875 - acc: 0.0018 23/88 [======>.......................] - ETA: 29s - loss: 5.8875 - acc: 0.0020 24/88 [=======>......................] - ETA: 29s - loss: 5.8874 - acc: 0.0020 25/88 [=======>......................] - ETA: 28s - loss: 5.8873 - acc: 0.0022 26/88 [=======>......................] - ETA: 28s - loss: 5.8872 - acc: 0.0021 27/88 [========>.....................] - ETA: 27s - loss: 5.8873 - acc: 0.0020 28/88 [========>.....................] - ETA: 27s - loss: 5.8874 - acc: 0.0020 29/88 [========>.....................] - ETA: 26s - loss: 5.8875 - acc: 0.0019 30/88 [=========>....................] - ETA: 26s - loss: 5.8874 - acc: 0.0021 31/88 [=========>....................] - ETA: 26s - loss: 5.8874 - acc: 0.0023 32/88 [=========>....................] - ETA: 25s - loss: 5.8872 - acc: 0.0022 33/88 [==========>...................] - ETA: 25s - loss: 5.8872 - acc: 0.0021 34/88 [==========>...................] - ETA: 25s - loss: 5.8872 - acc: 0.0021 35/88 [==========>...................] - ETA: 24s - loss: 5.8872 - acc: 0.0020 36/88 [===========>..................] - ETA: 24s - loss: 5.8872 - acc: 0.0020 37/88 [===========>..................] - ETA: 23s - loss: 5.8872 - acc: 0.0019 38/88 [===========>..................] - ETA: 23s - loss: 5.8872 - acc: 0.0019 39/88 [============>.................] - ETA: 22s - loss: 5.8873 - acc: 0.0018 40/88 [============>.................] - ETA: 22s - loss: 5.8873 - acc: 0.0020 41/88 [============>.................] - ETA: 21s - loss: 5.8873 - acc: 0.0021 42/88 [=============>................] - ETA: 21s - loss: 5.8872 - acc: 0.0020 43/88 [=============>................] - ETA: 20s - loss: 5.8873 - acc: 0.0020 44/88 [==============>...............] - ETA: 20s - loss: 5.8872 - acc: 0.0020 45/88 [==============>...............] - ETA: 19s - loss: 5.8873 - acc: 0.0019 46/88 [==============>...............] - ETA: 18s - loss: 5.8873 - acc: 0.0019 47/88 [===============>..............] - ETA: 18s - loss: 5.8873 - acc: 0.0020 48/88 [===============>..............] - ETA: 18s - loss: 5.8873 - acc: 0.0020 49/88 [===============>..............] - ETA: 17s - loss: 5.8873 - acc: 0.0021 50/88 [================>.............] - ETA: 17s - loss: 5.8873 - acc: 0.0020 51/88 [================>.............] - ETA: 16s - loss: 5.8873 - acc: 0.0020 52/88 [================>.............] - ETA: 16s - loss: 5.8873 - acc: 0.0021 53/88 [=================>............] - ETA: 15s - loss: 5.8873 - acc: 0.0024 54/88 [=================>............] - ETA: 15s - loss: 5.8873 - acc: 0.0023 55/88 [=================>............] - ETA: 14s - loss: 5.8874 - acc: 0.0024 56/88 [==================>...........] - ETA: 14s - loss: 5.8874 - acc: 0.0024 57/88 [==================>...........] - ETA: 13s - loss: 5.8873 - acc: 0.0023 58/88 [==================>...........] - ETA: 13s - loss: 5.8872 - acc: 0.0023 59/88 [===================>..........] - ETA: 13s - loss: 5.8872 - acc: 0.0023 60/88 [===================>..........] - ETA: 12s - loss: 5.8872 - acc: 0.0022 61/88 [===================>..........] - ETA: 12s - loss: 5.8871 - acc: 0.0023 62/88 [====================>.........] - ETA: 11s - loss: 5.8872 - acc: 0.0023 63/88 [====================>.........] - ETA: 11s - loss: 5.8873 - acc: 0.0022 64/88 [====================>.........] - ETA: 10s - loss: 5.8872 - acc: 0.0023 65/88 [=====================>........] - ETA: 10s - loss: 5.8872 - acc: 0.0023 66/88 [=====================>........] - ETA: 9s - loss: 5.8872 - acc: 0.0022 67/88 [=====================>........] - ETA: 9s - loss: 5.8873 - acc: 0.0023 68/88 [======================>.......] - ETA: 8s - loss: 5.8872 - acc: 0.0023 69/88 [======================>.......] - ETA: 8s - loss: 5.8872 - acc: 0.0023 70/88 [======================>.......] - ETA: 7s - loss: 5.8873 - acc: 0.0023 71/88 [=======================>......] - ETA: 7s - loss: 5.8873 - acc: 0.0024 72/88 [=======================>......] - ETA: 7s - loss: 5.8873 - acc: 0.0025 73/88 [=======================>......] - ETA: 6s - loss: 5.8873 - acc: 0.0026 74/88 [========================>.....] - ETA: 6s - loss: 5.8873 - acc: 0.0025 75/88 [========================>.....] - ETA: 5s - loss: 5.8874 - acc: 0.0025 76/88 [========================>.....] - ETA: 5s - loss: 5.8874 - acc: 0.0025 77/88 [=========================>....] - ETA: 4s - loss: 5.8875 - acc: 0.0024 78/88 [=========================>....] - ETA: 4s - loss: 5.8875 - acc: 0.0024 79/88 [=========================>....] - ETA: 3s - loss: 5.8875 - acc: 0.0024 80/88 [==========================>...] - ETA: 3s - loss: 5.8875 - acc: 0.0023 81/88 [==========================>...] - ETA: 3s - loss: 5.8875 - acc: 0.0023 82/88 [==========================>...] - ETA: 2s - loss: 5.8875 - acc: 0.0023 83/88 [===========================>..] - ETA: 2s - loss: 5.8876 - acc: 0.0023 84/88 [===========================>..] - ETA: 1s - loss: 5.8875 - acc: 0.0023 85/88 [===========================>..] - ETA: 1s - loss: 5.8875 - acc: 0.0023 86/88 [============================>.] - ETA: 0s - loss: 5.8874 - acc: 0.0023 87/88 [============================>.] - ETA: 0s - loss: 5.8875 - acc: 0.0023 88/88 [==============================] - 47s 538ms/step - loss: 5.8875 - acc: 0.0023 - val_loss: 5.8864 - val_acc: 0.0026 .... Epoch 18/500 1/88 [..............................] - ETA: 37s - loss: 5.7819 - acc: 0.0078 2/88 [..............................] - ETA: 36s - loss: 5.8187 - acc: 0.0039 3/88 [>.............................] - ETA: 36s - loss: 5.8373 - acc: 0.0052 4/88 [>.............................] - ETA: 36s - loss: 5.8308 - acc: 0.0059 5/88 [>.............................] - ETA: 36s - loss: 5.8144 - acc: 0.0063 6/88 [=>............................] - ETA: 35s - loss: 5.8129 - acc: 0.0052 7/88 [=>............................] - ETA: 36s - loss: 5.8067 - acc: 0.0045 8/88 [=>............................] - ETA: 35s - loss: 5.7977 - acc: 0.0039 9/88 [==>...........................] - ETA: 34s - loss: 5.7942 - acc: 0.0043 10/88 [==>...........................] - ETA: 34s - loss: 5.7885 - acc: 0.0047 11/88 [==>...........................] - ETA: 34s - loss: 5.7782 - acc: 0.0050 12/88 [===>..........................] - ETA: 33s - loss: 5.7755 - acc: 0.0052 13/88 [===>..........................] - ETA: 33s - loss: 5.7704 - acc: 0.0054 14/88 [===>..........................] - ETA: 32s - loss: 5.7718 - acc: 0.0056 15/88 [====>.........................] - ETA: 32s - loss: 5.7693 - acc: 0.0052 16/88 [====>.........................] - ETA: 31s - loss: 5.7715 - acc: 0.0049 17/88 [====>.........................] - ETA: 31s - loss: 5.7678 - acc: 0.0051 18/88 [=====>........................] - ETA: 30s - loss: 5.7648 - acc: 0.0048 19/88 [=====>........................] - ETA: 30s - loss: 5.7681 - acc: 0.0045 20/88 [=====>........................] - ETA: 30s - loss: 5.7695 - acc: 0.0047 21/88 [======>.......................] - ETA: 29s - loss: 5.7740 - acc: 0.0045 22/88 [======>.......................] - ETA: 28s - loss: 5.7734 - acc: 0.0043 23/88 [======>.......................] - ETA: 28s - loss: 5.7723 - acc: 0.0044 24/88 [=======>......................] - ETA: 28s - loss: 5.7775 - acc: 0.0042 25/88 [=======>......................] - ETA: 27s - loss: 5.7784 - acc: 0.0044 26/88 [=======>......................] - ETA: 27s - loss: 5.7762 - acc: 0.0045 27/88 [========>.....................] - ETA: 27s - loss: 5.7783 - acc: 0.0055 28/88 [========>.....................] - ETA: 26s - loss: 5.7791 - acc: 0.0064 29/88 [========>.....................] - ETA: 26s - loss: 5.7811 - acc: 0.0065 30/88 [=========>....................] - ETA: 25s - loss: 5.7860 - acc: 0.0063 31/88 [=========>....................] - ETA: 25s - loss: 5.7842 - acc: 0.0060 32/88 [=========>....................] - ETA: 25s - loss: 5.7830 - acc: 0.0059 33/88 [==========>...................] - ETA: 24s - loss: 5.7816 - acc: 0.0059 34/88 [==========>...................] - ETA: 24s - loss: 5.7783 - acc: 0.0060 35/88 [==========>...................] - ETA: 23s - loss: 5.7781 - acc: 0.0060 36/88 [===========>..................] - ETA: 23s - loss: 5.7763 - acc: 0.0061 37/88 [===========>..................] - ETA: 22s - loss: 5.7759 - acc: 0.0063 38/88 [===========>..................] - ETA: 22s - loss: 5.7772 - acc: 0.0062 39/88 [============>.................] - ETA: 21s - loss: 5.7813 - acc: 0.0060 40/88 [============>.................] - ETA: 21s - loss: 5.7812 - acc: 0.0061 41/88 [============>.................] - ETA: 20s - loss: 5.7818 - acc: 0.0059 42/88 [=============>................] - ETA: 20s - loss: 5.7841 - acc: 0.0058 43/88 [=============>................] - ETA: 19s - loss: 5.7826 - acc: 0.0056 44/88 [==============>...............] - ETA: 19s - loss: 5.7819 - acc: 0.0057 45/88 [==============>...............] - ETA: 18s - loss: 5.7823 - acc: 0.0056 46/88 [==============>...............] - ETA: 18s - loss: 5.7835 - acc: 0.0054 47/88 [===============>..............] - ETA: 17s - loss: 5.7845 - acc: 0.0055 48/88 [===============>..............] - ETA: 17s - loss: 5.7849 - acc: 0.0055 49/88 [===============>..............] - ETA: 17s - loss: 5.7845 - acc: 0.0056 50/88 [================>.............] - ETA: 16s - loss: 5.7842 - acc: 0.0055 51/88 [================>.............] - ETA: 16s - loss: 5.7845 - acc: 0.0054 52/88 [================>.............] - ETA: 15s - loss: 5.7839 - acc: 0.0056 53/88 [=================>............] - ETA: 15s - loss: 5.7820 - acc: 0.0056 54/88 [=================>............] - ETA: 14s - loss: 5.7832 - acc: 0.0058 55/88 [=================>............] - ETA: 14s - loss: 5.7845 - acc: 0.0058 56/88 [==================>...........] - ETA: 13s - loss: 5.7845 - acc: 0.0057 57/88 [==================>...........] - ETA: 13s - loss: 5.7837 - acc: 0.0058 58/88 [==================>...........] - ETA: 13s - loss: 5.7835 - acc: 0.0057 59/88 [===================>..........] - ETA: 12s - loss: 5.7830 - acc: 0.0057 60/88 [===================>..........] - ETA: 12s - loss: 5.7821 - acc: 0.0056 61/88 [===================>..........] - ETA: 11s - loss: 5.7831 - acc: 0.0055 62/88 [====================>.........] - ETA: 11s - loss: 5.7834 - acc: 0.0055 63/88 [====================>.........] - ETA: 10s - loss: 5.7836 - acc: 0.0055 64/88 [====================>.........] - ETA: 10s - loss: 5.7836 - acc: 0.0054 65/88 [=====================>........] - ETA: 9s - loss: 5.7817 - acc: 0.0056 66/88 [=====================>........] - ETA: 9s - loss: 5.7820 - acc: 0.0057 67/88 [=====================>........] - ETA: 9s - loss: 5.7842 - acc: 0.0057 68/88 [======================>.......] - ETA: 8s - loss: 5.7828 - acc: 0.0057 69/88 [======================>.......] - ETA: 8s - loss: 5.7824 - acc: 0.0057 70/88 [======================>.......] - ETA: 7s - loss: 5.7838 - acc: 0.0056 71/88 [=======================>......] - ETA: 7s - loss: 5.7843 - acc: 0.0055 72/88 [=======================>......] - ETA: 6s - loss: 5.7831 - acc: 0.0055 73/88 [=======================>......] - ETA: 6s - loss: 5.7831 - acc: 0.0056 74/88 [========================>.....] - ETA: 6s - loss: 5.7841 - acc: 0.0055 75/88 [========================>.....] - ETA: 5s - loss: 5.7837 - acc: 0.0054 76/88 [========================>.....] - ETA: 5s - loss: 5.7843 - acc: 0.0053 77/88 [=========================>....] - ETA: 4s - loss: 5.7862 - acc: 0.0053 78/88 [=========================>....] - ETA: 4s - loss: 5.7871 - acc: 0.0053 79/88 [=========================>....] - ETA: 3s - loss: 5.7874 - acc: 0.0052 80/88 [==========================>...] - ETA: 3s - loss: 5.7867 - acc: 0.0054 81/88 [==========================>...] - ETA: 2s - loss: 5.7859 - acc: 0.0054 82/88 [==========================>...] - ETA: 2s - loss: 5.7850 - acc: 0.0054 83/88 [===========================>..] - ETA: 2s - loss: 5.7850 - acc: 0.0055 84/88 [===========================>..] - ETA: 1s - loss: 5.7846 - acc: 0.0054 85/88 [===========================>..] - ETA: 1s - loss: 5.7831 - acc: 0.0054 86/88 [============================>.] - ETA: 0s - loss: 5.7827 - acc: 0.0055 87/88 [============================>.] - ETA: 0s - loss: 5.7827 - acc: 0.0054 88/88 [==============================] - 46s 524ms/step - loss: 5.7836 - acc: 0.0055 - val_loss: 5.7613 - val_acc: 0.0059

When I'm trying to evaluate my model with pre-computed weights like this:

model.load_weights('../checkpoints/small_model_epoch_5.h5')
score = model.evaluate_generator(generator=test_generator.generate(batch_size, num_classes),
                         steps=test_generator.get_num_samples() / batch_size)
print(score)

I'm receiving poor result:

loss                 accuracy
[15.136031714352695, 0.01455965909090909]

UPD. total num games: 179689

Until this point, all models' training was going ok, like in the book. BTW thank you for the great book and will appreciate help with this problem :)

maxpumperla commented 4 years ago

@y-kkky I thanks for reporting this and great to hear you enjoy the book. None of what you see is actually a "problem" as such. For the book we mostly used really bad hardware, so as not to discourage readers that don't have fancy GPUs. If your training on 100 games is fast, that might not be an issue.

It might be interesting to see how many games are actually downloaded / how many positions you have in your dataset. From the epochs you posted it looks like there are only 88 samples in your data set, which seems too low. you could try to increase the number of games to see what happens. Training on such few data points, I wouldn't expect much in terms of learning.

P.s.: if you check the Manning liveBook page, you'll see that it has been reported many times that "accuracy was incorrectly reported" in chapter 7. that is correct.

y-kkky commented 4 years ago

@maxpumperla thanks for answering! About samples: it is 88 when I'm using fit_generator, when I'm not using generator (consolidate_games instead), it reports sample number correctly (smth near 12288 as in the book). Also for some reason test data is larger than train data (12288 and 11xxx accordingly), some mess with Sampler I guess but did not find out what exactly.

And what about precomputed weights in small_model_epoch_5.h5 - even if I have not enough positions in the dataset, the model is the same so it should predict with accuracy, higher than 0.0145.

Looks like data is in the wrong format for some reason (it is my only guess), but I'm using exactly the same encoder as you (OneplaneEncoder)

Details import numpy as np from dlgo.encoders.base import Encoder from dlgo.goboard import Point class OnePlaneEncoder(Encoder): def __init__(self, board_size): self.board_width, self.board_height = board_size self.num_planes = 1 def name(self): return 'oneplane' def encode(self, game_state): board_matrix = np.zeros(self.shape()) next_player = game_state.next_player for r in range(self.board_height): for c in range(self.board_width): p = Point(row=r + 1, col=c + 1) go_string = game_state.board.get_go_string(p) if go_string is None: continue if go_string.color == next_player: board_matrix[0, r, c] = 1 else: board_matrix[0, r, c] = -1 return board_matrix def encode_point(self, point): return self.board_width * (point.row - 1) + (point.col - 1) def decode_point_index(self, index): row = index // self.board_width col = index % self.board_width return Point(row=row + 1, col=col + 1) def num_points(self): return self.board_width * self.board_height def shape(self): return self.num_planes, self.board_height, self.board_width def create(board_size): return OnePlaneEncoder(board_size)
maxpumperla commented 4 years ago

@y-kkky hmm, strange. it's very difficult for me to figure out how to help you in this situation. some things can indeed change. upstream data (unlikely), some format changes that mess up Sampler (reasonable), or maybe newer dependencies of Python libs (we have a lot of problems since TF 2 came out for instance).

@macfergus is this something you'd have the capacity to check out right now? did we report the wrong results maybe? thanks in any case.

macfergus commented 4 years ago

Hi @y-kkky, try modifying this line:

generator = processor.load_go_data('train', num_games, use_generator=True)

To something like this:

generator = processor.load_go_data('train', num_games, num_samples=100000, use_generator=True)

You can adjust the num_samples param to keep more or less data from the source files. The data processor does down-sampling on the input (for the performance reasons Max mentioned), seems like its default is down-sampling a little too aggressively 🤔

I think for the example output in the book, we used a much higher number of samples.

y-kkky commented 4 years ago

@macfergus thank you! that explains a lot.

And what about small_modelepoch{epoch}.h5 in master - with what combination of encoder and model they were generated? I'm trying on OneplaneEncoder I've mentioned earlier with the small model from latest master.

BTW thank you for your help, I thought that I'm doing smth wrong and that I should solve the problem now to proceed to the next chapter, but now I think I'll just try to proceed cause after increasing the number of samples it started learning better. Thanks 👍