felixbur / nkululeko

Machine learning speaker characteristics
MIT License
26 stars 4 forks source link

add huberts #57

Closed bagustris closed 10 months ago

bagustris commented 10 months ago

This adds five variants of Hubert: hubert, hubert_ft, hubert_large, hubert_xlarge, and hubert_xlarge_ft (Reference). Here are the results of the first four variants on RAVDESS dataset. This PR is to tackle #56.

# Hubert (base)
UG modelrunner: run: 0 epoch: 0: result: test: 0.742 UAR
DEBUG modelrunner: plotting confusion matrix to train_test_dev_svm_hubert__0_000_cnf
DEBUG runmanager: value for measure not found, using default: uar
DEBUG reporter: labels: ['angry' 'happy' 'neutral' 'sad']
DEBUG reporter: result per class (F1 score): [0.909, 0.8, 0.4, 0.849]
WARNING experiment: Save experiment: Can't pickle local object: Serialization of parametrized modules is only supported through state_dict(). See:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training
DEBUG experiment: Done, used 694.146 seconds
DONE
---
# Hubert_ft (finetuning)

DEBUG modelrunner: run: 0 epoch: 0: result: test: 0.891 UAR
DEBUG modelrunner: plotting confusion matrix to train_test_dev_svm_hubert__0_000_cnf
DEBUG runmanager: value for measure not found, using default: uar
DEBUG reporter: labels: ['angry' 'happy' 'neutral' 'sad']
DEBUG reporter: result per class (F1 score): [0.935, 0.814, 0.842, 0.892]
WARNING experiment: Save experiment: Can't pickle local object: Serialization of parametrized modules is only supported through state_dict(). See:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training
DEBUG experiment: Done, used 1998.011 seconds
DONE
-----
# Hubert_large

DEBUG modelrunner: run: 0 epoch: 0: result: test: 0.914 UAR
DEBUG modelrunner: plotting confusion matrix to train_test_dev_svm_hubert_large_ll60k__0_000_cnf
DEBUG runmanager: value for measure not found, using default: uar
DEBUG reporter: labels: ['angry' 'happy' 'neutral' 'sad']
DEBUG reporter: result per class (F1 score): [0.984, 0.881, 0.882, 0.882]
WARNING experiment: Save experiment: Can't pickle local object: Serialization of parametrized modules is only supported through state_dict(). See:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training
DEBUG experiment: Done, used 1867.607 seconds
DONE
---
# Hubert_xlarge
DEBUG modelrunner: run: 0 epoch: 0: result: test: 0.852 UAR
DEBUG modelrunner: plotting confusion matrix to train_test_dev_svm_hubert_xlarge__0_000_cnf
DEBUG runmanager: value for measure not found, using default: uar
DEBUG reporter: labels: ['angry' 'happy' 'neutral' 'sad']
DEBUG reporter: result per class (F1 score): [0.899, 0.821, 0.788, 0.879]
WARNING experiment: Save experiment: Can't pickle local object: Serialization of parametrized modules is only supported through state_dict(). See:
https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-a-general-checkpoint-for-inference-and-or-resuming-training
DEBUG experiment: Done, used 4410.192 seconds

I also modified the wav2vec feature file to adopt the style introduced in this PR, so it will be able to receive variants of wav2vec2 (not done yet).

Example of INI file for Ravdess (nkululeko.nkululeko)

[EXP]
root = ./
name = results/exp_ravdess_hubert
runs = 1
epochs = 1
save = True
[DATA]
databases = ['train', 'test', 'dev']
train = ./data/ravdess/ravdess_train.csv
train.type = csv
train.absolute_path = False
train.split_strategy = train
dev = ./data/ravdess/ravdess_dev.csv
dev.type = csv
dev.absolute_path = False
dev.split_strategy = train
test = ./data/ravdess/ravdess_test.csv
test.type = csv
test.absolute_path = False
test.split_strategy = test
target = emotion
labels = ['angry', 'happy', 'neutral', 'sad']
[FEATS]
type = ['hubert_xlarge']
scale = standard
[MODEL]
type = svm