search

TmacMai / multimodal-fusion

Multimodal Fusion, Multimodal Sentiment Analysis
20 stars 0 forks source link

How is F1 score calculated in Divide, Conquer and Combine... paper? #2

Open yiwang454 opened 3 years ago

yiwang454 commented 3 years ago

Hi Sijie @TmacMai,

I just started working on multimodal learning using CMU-MOSEI, and I was wondering how the F1 scores evaluated on CMU-MOSEI dataset was calculated? Was it done by calculating F1 score for each emotion first and getting an average afterwards?

Thank you for your reply in advance

yiwang454 commented 3 years ago

Also, I'm wondering how you dealt with multi-class classification taking into account that some of the utterances are labelled with several annotations (e.g. some utterances can have emotion labels like ['0.56', '0.11', '0.11', '0.22', '0.00', '0.00'])

TmacMai commented 3 years ago

Hi Sijie @TmacMai,

I just started working on multimodal learning using CMU-MOSEI, and I was wondering how the F1 scores evaluated on CMU-MOSEI dataset was calculated? Was it done by calculating F1 score for each emotion first and getting an average afterwards?

Thank you for your reply in advance

Hi, thanks for your attention to our work. The calculation of F1 scores for CMU-MOSEI dataset is shown in https://github.com/TmacMai/ARGF_multimodal_fusion, where we place the code and the data for our AAAI-20 paper. You can easily read the code or run it directly.

TmacMai commented 3 years ago

Also, I'm wondering how you dealt with multi-class classification taking into account that some of the utterances are labelled with several annotations (e.g. some utterances can have emotion labels like ['0.56', '0.11', '0.11', '0.22', '0.00', '0.00'])

It seems that the sum of all the emotional scores is equal to 1? This is a much more complex case. You can still use the softmax function on the output, but corss-entropy is not suggested. You might apply a MSE loss to update your model? Considering that the label of each single emotion is not binary (0 or 1), I would suggest you to use regression metrics (MAE, Corr, etc.) to evaluate your model. If the label is binary, one common way to deal with multi-class classification is to decouple the prediction of each emotion into a binary classification task (such as the dim of the output can be set to n*2 where n is the number of emotions, and use each 2 dimension to represent the probability of a certain kind of emotion). I'm not sure whether I have make myself clear. Feel free to discuss with me if you have any more questions.

TmacMai commented 3 years ago

Also, I'm wondering how you dealt with multi-class classification taking into account that some of the utterances are labelled with several annotations (e.g. some utterances can have emotion labels like ['0.56', '0.11', '0.11', '0.22', '0.00', '0.00'])

It seems that the sum of all the emotional scores is equal to 1? This is a much more complex case. You can still use the softmax function on the output, but corss-entropy is not suggested. You might apply a MSE loss to update your model? Considering that the label of each single emotion is not binary (0 or 1), I would suggest you to use regression metrics (MAE, Corr, etc.) to evaluate your model. If the label is binary, one common way to deal with multi-class classification is to decouple the prediction of each emotion into a binary classification task (such as the dim of the output can be set to n*2 where n is the number of emotions, and use each 2 dimension to represent the probability of a certain kind of emotion). I'm not sure whether I have make myself clear. Feel free to discuss with me if you have any more questions.