search

mdswyz / DiCMoR

An official implementation of "Distribution-Consistent Modal Recovering for Incomplete Multimodal Learning" in PyTorch. (ICCV 2023)
Apache License 2.0
25 stars 2 forks source link

How to replicate paper results? #5

Open artemisp opened 5 months ago

artemisp commented 5 months ago

Attempting to retrain the model with MR=0.0 results to significantly lower performance (F1 .55, compared to .85 in the paper).

Moreover, even loading the pretrained model with MR=0.0 yields about ~5 points lower F1, and similarly, as MR increases the performance gap between reported scores and replicated scores increases up to 20 points in F1 for MR=.7.

Could you please provide some guidance on what could be the issue?

Thank you!

yamseyoung commented 3 days ago

I got the same result (acc2=.58) as seen in the generated result.csv file, but I noticed that in the train log, the final performance on test data is as the following: - MMSA [INFO] - TEST-(dicmor) >> Acc_2: 0.8155 F1_score: 0.8148 Acc_7: 0.4052 MAE: 0.8233 Loss: 0.8227

There seems to be a discrepency in these two results. Could the authors kindly explains this?