search

thuiar / MMSA

MMSA is a unified framework for Multimodal Sentiment Analysis.
MIT License
634 stars 104 forks source link

MulT-MOSEI #68

Open efthymisgeo opened 1 year ago

efthymisgeo commented 1 year ago

Hello. I have tried to reproduce MulT (both aligned and unaligned) based on your code and I cannot match the (average) results you have reported in the paper. I have tried several seeds but still i am not able to match your performance. I tested the code in an Ubuntu20.04 machine and in particular within a conda environment (python 3.10) with all the corresponding requirements satisfied. Finally, i have cloned the repo (no editing) instead of using the command line tool.

My main concern here is that _acc2 and F1 metrics are almost 1% less than yours. This result is of course statistically significant (the corresponding std's are shown after the comma char). Do you have any good reason why am I getting that much of a difference? Could it be aligned /unaligned, any missing hyperparameters, pytorch version etc (mine is 1.13) ??

Model Non0_acc_2 Non0_F1_score Mult_acc_5 Mult_acc_7 MAE Corr Data Setting
mult-results.md (yours) 84.63 84.52 54.18 52.84 55.93 73.31 Unaligned
mult (myself) 83.69, 0.25 83.64, 0.21 53.94, 0.42 52.66, 0.37 56.08, 0.32 73.17, 0.51 Unaligned

Thanks in advance.

py-rgb commented 8 months ago

Due to variations in experimental environments, it is normal for experimental results to fluctuate. A decrease of 1.13% is within an acceptable range. MMSA provides config_tune to assist you in selecting the best results specific to your experimental environment.