search

jbdel / MOSEI_UMONS

A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis
MIT License
118 stars 35 forks source link

Accuracy mismatch #22

Closed cometome closed 2 years ago

cometome commented 2 years ago

Hi, I found this resource as an important starting point for multimodal data analysis in emotion recognition. I used this repository and computed the accuracy. Accuracy is coming similar, but while computed emotion wise recognition accuracy, i observed few emotions are never been detected correctly. The F1 Score is coming very low (probably due to unbalanced data distribution). Could you please help me to understand where I am doing mistake?

hadikachmar3 commented 2 years ago

Hello,

I also have the same problem. If someone can help?

jbdel commented 2 years ago

Hello.

If I recall correctly, yes: the accuracy for some emotions are very low. Please note that the scores reported in the papers are the f1-weighted scores, that can be very different than f1-micro or macro.

Also the paper states the scores from our submissions for the MOSEI-challenge, that were obtained using some class-weighting during the loss (see here in another repo of mine here for example: https://github.com/jbdel/modulated_fusion_transformer/blob/main/utils/compute_args.py#L18). You might want to play around with that as well.

This repo is the baseline for all our experiments, though you should be able to replicate the scores in the readme.

Good luck !

JB