Thunfischpirat
commented
1 year ago Retrain the model and see what happens. Make sure to train on "TRAIN" split and not on speaker names. In transformer_model.py change
96 train_set = ["george"]
97 val_set = ["jackson", "lucas", "nicolas", "yweweler", "theo"]to
96 train_set = "TRAIN"
97 val_set = "DEV"
/Users/introvertuoso/python3.10_venv/bin/python /Users/introvertuoso/PycharmProjects/SpokenDigits/model_neural/classification_report.py Using: 'cpu' as device for report. /Users/introvertuoso/python3.10_venv/lib/python3.9/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment. warnings.warn('Lazy modules are a new feature under heavy development '
Confusion matrix of train-set: tensor([[ 22, 0, 0, 0, 0, 0, 170, 0, 8, 0], [ 0, 23, 0, 0, 6, 3, 162, 0, 5, 1], [ 0, 0, 9, 0, 0, 0, 191, 0, 0, 0], [ 0, 0, 7, 3, 0, 0, 190, 0, 0, 0], [ 0, 0, 0, 0, 28, 2, 165, 0, 5, 0], [ 0, 0, 0, 0, 2, 53, 129, 0, 16, 0], [ 0, 0, 0, 0, 6, 0, 180, 0, 14, 0], [ 0, 0, 1, 0, 16, 0, 162, 5, 16, 0], [ 0, 0, 0, 0, 0, 0, 168, 0, 32, 0], [ 1, 16, 11, 4, 0, 26, 105, 3, 13, 21]])
Classification report: train-set: class precision recall f1-score 0 0.96 0.11 0.20
1 0.59 0.12 0.19
2 0.32 0.05 0.08
3 0.43 0.01 0.03
4 0.48 0.14 0.22
5 0.63 0.26 0.37
6 0.11 0.90 0.20
7 0.62 0.03 0.05
8 0.29 0.16 0.21
9 0.95 0.10 0.19
accuracy: 0.19
Confusion matrix of dev-set: tensor([[ 0, 0, 0, 0, 0, 0, 47, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 45, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 46, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 54, 0, 0, 0], [ 0, 0, 0, 0, 11, 0, 36, 0, 7, 0], [ 0, 2, 0, 0, 0, 9, 31, 0, 8, 0], [ 0, 0, 0, 0, 0, 0, 54, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 53, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 43, 0, 1, 0], [ 0, 2, 0, 1, 0, 8, 35, 0, 2, 2]])
Classification report: dev-set: class precision recall f1-score 0 0.00 0.00 0.00
1 0.00 0.00 0.00
2 0.00 0.00 0.00
3 0.00 0.00 0.00
4 1.00 0.20 0.34
5 0.53 0.18 0.27
6 0.12 1.00 0.22
7 0.00 0.00 0.00
8 0.06 0.02 0.03
9 1.00 0.04 0.08
accuracy: 0.14
Confusion matrix of test-set: tensor([[ 0, 0, 0, 0, 0, 0, 53, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 55, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 54, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 46, 0, 0, 0], [ 0, 0, 0, 0, 10, 0, 31, 0, 5, 0], [ 0, 2, 0, 0, 0, 6, 36, 0, 6, 0], [ 0, 0, 0, 0, 0, 0, 46, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 47, 0, 0, 0], [ 0, 0, 0, 0, 0, 0, 55, 0, 1, 0], [ 0, 7, 0, 0, 0, 8, 28, 0, 2, 5]])
Classification report: test-set: class precision recall f1-score 0 0.00 0.00 0.00
1 0.00 0.00 0.00
2 0.00 0.00 0.00
3 0.00 0.00 0.00
4 1.00 0.22 0.36
5 0.43 0.12 0.19
6 0.10 1.00 0.19
7 0.00 0.00 0.00
8 0.07 0.02 0.03
9 1.00 0.10 0.18
accuracy: 0.15
Process finished with exit code 0