search

felixbur / nkululeko

Machine learning speaker characteristics
MIT License
26 stars 4 forks source link

MLP model didn't report the best model but last epoch #130

Closed bagustris closed 2 weeks ago

bagustris commented 3 weeks ago

The current MLP didn't report the best performed model but instead the last epoch,

...
DEBUG model: value for device not found, using default: cuda
DEBUG model: using layers {'l1':128, 'l2':64}
DEBUG model: value for learning_rate not found, using default: 0.0001
DEBUG model: value for num_workers not found, using default: 5
DEBUG modelrunner: run: 0 epoch: 0: result: test: 0.500 UAR
DEBUG modelrunner: run: 0 epoch: 1: result: test: 0.500 UAR
DEBUG modelrunner: run: 0 epoch: 2: result: test: 0.500 UAR
DEBUG modelrunner: run: 0 epoch: 3: result: test: 0.500 UAR
DEBUG modelrunner: run: 0 epoch: 4: result: test: 0.503 UAR
DEBUG modelrunner: run: 0 epoch: 5: result: test: 0.509 UAR
DEBUG modelrunner: run: 0 epoch: 6: result: test: 0.516 UAR
DEBUG modelrunner: run: 0 epoch: 7: result: test: 0.518 UAR
DEBUG modelrunner: run: 0 epoch: 8: result: test: 0.521 UAR
DEBUG modelrunner: run: 0 epoch: 9: result: test: 0.520 UAR
DEBUG modelrunner: run: 0 epoch: 10: result: test: 0.523 UAR
DEBUG modelrunner: run: 0 epoch: 11: result: test: 0.522 UAR
DEBUG modelrunner: run: 0 epoch: 12: result: test: 0.526 UAR
DEBUG modelrunner: run: 0 epoch: 13: result: test: 0.523 UAR
DEBUG modelrunner: run: 0 epoch: 14: result: test: 0.521 UAR
DEBUG modelrunner: run: 0 epoch: 15: result: test: 0.518 UAR
DEBUG modelrunner: run: 0 epoch: 16: result: test: 0.523 UAR
DEBUG modelrunner: run: 0 epoch: 17: result: test: 0.526 UAR
DEBUG modelrunner: run: 0 epoch: 18: result: test: 0.522 UAR
DEBUG modelrunner: run: 0 epoch: 19: result: test: 0.521 UAR
DEBUG modelrunner: plotting confusion matrix to train_dev_mlp_os_64-128_scale-standard_0_019_cnf
DEBUG reporter: epoch: 19, UAR: .52, (+-.508/.534), ACC: .961

In the above case, the best UAR score is in epoch 12, but reporter report epoch 19 with its score.

It is necessary to report the best-performed model (along with its epoch number, I guess), although the user can manually inspect in which epoch the score (UAR) is highest.

In addition, I suggest printing the loss for every epoch since the best model also usually is picked from the lowest loss instead of the highest performance score.