search

napulen / phd_thesis

Automatic Roman numeral analysis in symbolic music representations.
1 stars 1 forks source link

Experiments with quality metrics #8

Closed napulen closed 3 years ago

napulen commented 3 years ago

I just ran an experiment where I filter out all the data that has a bad quality score from my metrics:

This is the output

Epoch 1/10
194/194 [==============================] - 8s 32ms/step - loss: 3.8248 - categorical_accuracy: 0.1534 - val_loss: 2.8037 - val_categorical_accuracy: 0.2499
Epoch 2/10
194/194 [==============================] - 6s 29ms/step - loss: 2.5797 - categorical_accuracy: 0.3091 - val_loss: 2.6528 - val_categorical_accuracy: 0.2896
Epoch 3/10
194/194 [==============================] - 6s 29ms/step - loss: 2.4970 - categorical_accuracy: 0.3387 - val_loss: 2.5727 - val_categorical_accuracy: 0.3097
Epoch 4/10
194/194 [==============================] - 6s 28ms/step - loss: 2.3853 - categorical_accuracy: 0.3575 - val_loss: 2.5171 - val_categorical_accuracy: 0.3302
Epoch 5/10
194/194 [==============================] - 6s 29ms/step - loss: 2.3226 - categorical_accuracy: 0.3738 - val_loss: 2.4543 - val_categorical_accuracy: 0.3470
Epoch 6/10
194/194 [==============================] - 6s 29ms/step - loss: 2.2792 - categorical_accuracy: 0.3934 - val_loss: 2.4040 - val_categorical_accuracy: 0.3704
Epoch 7/10
194/194 [==============================] - 6s 29ms/step - loss: 2.2074 - categorical_accuracy: 0.4116 - val_loss: 2.3618 - val_categorical_accuracy: 0.3792
Epoch 8/10
194/194 [==============================] - 6s 28ms/step - loss: 2.1539 - categorical_accuracy: 0.4245 - val_loss: 2.3298 - val_categorical_accuracy: 0.3918
Epoch 9/10
194/194 [==============================] - 6s 29ms/step - loss: 2.1484 - categorical_accuracy: 0.4269 - val_loss: 2.3080 - val_categorical_accuracy: 0.3908
Epoch 10/10
194/194 [==============================] - 6s 28ms/step - loss: 2.0946 - categorical_accuracy: 0.4390 - val_loss: 2.2846 - val_categorical_accuracy: 0.3991

The performance is better than when I use all the data

Epoch 1/10
210/210 [==============================] - 8s 31ms/step - loss: 3.8163 - categorical_accuracy: 0.1539 - val_loss: 2.7574 - val_categorical_accuracy: 0.2606
Epoch 2/10
210/210 [==============================] - 6s 29ms/step - loss: 2.5715 - categorical_accuracy: 0.3047 - val_loss: 2.6707 - val_categorical_accuracy: 0.2816
Epoch 3/10
210/210 [==============================] - 6s 29ms/step - loss: 2.5065 - categorical_accuracy: 0.3287 - val_loss: 2.6090 - val_categorical_accuracy: 0.2990
Epoch 4/10
210/210 [==============================] - 6s 29ms/step - loss: 2.4309 - categorical_accuracy: 0.3389 - val_loss: 2.5479 - val_categorical_accuracy: 0.3188
Epoch 5/10
210/210 [==============================] - 6s 29ms/step - loss: 2.3745 - categorical_accuracy: 0.3582 - val_loss: 2.4903 - val_categorical_accuracy: 0.3379
Epoch 6/10
210/210 [==============================] - 6s 29ms/step - loss: 2.3164 - categorical_accuracy: 0.3728 - val_loss: 2.4329 - val_categorical_accuracy: 0.3531
Epoch 7/10
210/210 [==============================] - 6s 29ms/step - loss: 2.2651 - categorical_accuracy: 0.3940 - val_loss: 2.4015 - val_categorical_accuracy: 0.3670
Epoch 8/10
210/210 [==============================] - 6s 29ms/step - loss: 2.2273 - categorical_accuracy: 0.4018 - val_loss: 2.3690 - val_categorical_accuracy: 0.3706
Epoch 9/10
210/210 [==============================] - 6s 29ms/step - loss: 2.2036 - categorical_accuracy: 0.4083 - val_loss: 2.3477 - val_categorical_accuracy: 0.3796
Epoch 10/10
210/210 [==============================] - 6s 29ms/step - loss: 2.2005 - categorical_accuracy: 0.4113 - val_loss: 2.3279 - val_categorical_accuracy: 0.3824

This is good news. Less data of higher quality is better. Also, my metrics seem to work.

Big caveat on this experiment: I applied the filter to the validation portion too. In practice, I shouldn't filter the validation data in this way at all.

Trying out the filter in training-only data is going to be the next step.

Also, this was predicting roman numerals from the Micchi19 representation. I need to verify how well the intervallic input representation does.

napulen commented 3 years ago

The filtered (including validation) interval representation

Epoch 1/10
194/194 [==============================] - 8s 32ms/step - loss: 3.7976 - categorical_accuracy: 0.1476 - val_loss: 2.6604 - val_categorical_accuracy: 0.3047
Epoch 2/10
194/194 [==============================] - 6s 29ms/step - loss: 2.4368 - categorical_accuracy: 0.3671 - val_loss: 2.4694 - val_categorical_accuracy: 0.3586
Epoch 3/10
194/194 [==============================] - 6s 29ms/step - loss: 2.3088 - categorical_accuracy: 0.3876 - val_loss: 2.3679 - val_categorical_accuracy: 0.3765
Epoch 4/10
194/194 [==============================] - 6s 29ms/step - loss: 2.2002 - categorical_accuracy: 0.4115 - val_loss: 2.3182 - val_categorical_accuracy: 0.3956
Epoch 5/10
194/194 [==============================] - 6s 29ms/step - loss: 2.1624 - categorical_accuracy: 0.4207 - val_loss: 2.2839 - val_categorical_accuracy: 0.4031
Epoch 6/10
194/194 [==============================] - 6s 29ms/step - loss: 2.1479 - categorical_accuracy: 0.4262 - val_loss: 2.2577 - val_categorical_accuracy: 0.4060
Epoch 7/10
194/194 [==============================] - 6s 29ms/step - loss: 2.0991 - categorical_accuracy: 0.4362 - val_loss: 2.2471 - val_categorical_accuracy: 0.4066
Epoch 8/10
194/194 [==============================] - 6s 29ms/step - loss: 2.0593 - categorical_accuracy: 0.4416 - val_loss: 2.2297 - val_categorical_accuracy: 0.4130
Epoch 9/10
194/194 [==============================] - 6s 29ms/step - loss: 2.0680 - categorical_accuracy: 0.4394 - val_loss: 2.2189 - val_categorical_accuracy: 0.4090
Epoch 10/10
194/194 [==============================] - 6s 29ms/step - loss: 2.0387 - categorical_accuracy: 0.4467 - val_loss: 2.2102 - val_categorical_accuracy: 0.4148

It outperforms the micchi19 in predicting roman numeral labels.

napulen commented 3 years ago

The same experiment as before, except that only the training data has been filtered:

Epoch 1/10
194/194 [==============================] - 6s 24ms/step - loss: 3.7976 - categorical_accuracy: 0.1476 - val_loss: 2.6734 - val_categorical_accuracy: 0.3002
Epoch 2/10
194/194 [==============================] - 4s 22ms/step - loss: 2.4368 - categorical_accuracy: 0.3671 - val_loss: 2.4880 - val_categorical_accuracy: 0.3526
Epoch 3/10
194/194 [==============================] - 4s 22ms/step - loss: 2.3088 - categorical_accuracy: 0.3876 - val_loss: 2.3916 - val_categorical_accuracy: 0.3702
Epoch 4/10
194/194 [==============================] - 4s 22ms/step - loss: 2.2002 - categorical_accuracy: 0.4115 - val_loss: 2.3458 - val_categorical_accuracy: 0.3884
Epoch 5/10
194/194 [==============================] - 4s 22ms/step - loss: 2.1624 - categorical_accuracy: 0.4207 - val_loss: 2.3129 - val_categorical_accuracy: 0.3964
Epoch 6/10
194/194 [==============================] - 4s 22ms/step - loss: 2.1479 - categorical_accuracy: 0.4262 - val_loss: 2.2891 - val_categorical_accuracy: 0.3998
Epoch 7/10
194/194 [==============================] - 4s 21ms/step - loss: 2.0991 - categorical_accuracy: 0.4362 - val_loss: 2.2776 - val_categorical_accuracy: 0.4002
Epoch 8/10
194/194 [==============================] - 4s 21ms/step - loss: 2.0593 - categorical_accuracy: 0.4416 - val_loss: 2.2614 - val_categorical_accuracy: 0.4081
Epoch 9/10
194/194 [==============================] - 4s 21ms/step - loss: 2.0680 - categorical_accuracy: 0.4394 - val_loss: 2.2517 - val_categorical_accuracy: 0.4052
Epoch 10/10
194/194 [==============================] - 4s 21ms/step - loss: 2.0387 - categorical_accuracy: 0.4467 - val_loss: 2.2433 - val_categorical_accuracy: 0.4091