search

thuiar / MMSA

MMSA is a unified framework for Multimodal Sentiment Analysis.
MIT License
642 stars 104 forks source link

Wrong input order of f1 score calculation? #17

Closed XpastaX closed 3 years ago

XpastaX commented 3 years ago

Hi, I am a bit confused about the f1 score calculation of your code. Based on your code: https://github.com/thuiar/MMSA/blob/bf88dfc8073193a7566dadeae7d6de82f1e86a56/trains/singleTask/BERT_MAG.py#L130 and https://github.com/thuiar/MMSA/blob/bf88dfc8073193a7566dadeae7d6de82f1e86a56/utils/metricsTop.py#L111 f1 score is calculated with sk.learn.metrics.f1_score(pred,true), but in the official document, it should be calculated by sk.learn.metrics.f1_score(true,pred). The ground truth should be first. Is this a mistake? Does it affect the final result of your paper?

sklearn official document: https://scikit-learn.org/0.21/modules/generated/sklearn.metrics.f1_score.html

iyuge2 commented 3 years ago

@JiudingYang Oh, It is a mistake. If the average='macro' or average='micro', there is no difference in two ways. However, it will affect the final result in the average='weighted'.

In our implementation, these codes are based on the https://github.com/yaohungt/Multimodal-Transformer/blob/0efe31b75e9f59ab9de6f1bd08ed6cb5d1129e68/src/eval_metrics.py#L46

Thanks for your kind reminder. Sorry, we didn't notice this problem before. We will re-run all results again and update them in results/result-stat.md very soon.

iyuge2 commented 3 years ago

@JiudingYang Hi，we have updated all results. Besides, we update some parameters according to issue #10. Thanks you~