Open efthymisgeo opened 1 year ago
Due to variations in experimental environments, it is normal for experimental results to fluctuate. A decrease of 1.13% is within an acceptable range. MMSA provides config_tune to assist you in selecting the best results specific to your experimental environment.
Hello. I have tried to reproduce MulT (both aligned and unaligned) based on your code and I cannot match the (average) results you have reported in the paper. I have tried several seeds but still i am not able to match your performance. I tested the code in an Ubuntu20.04 machine and in particular within a conda environment (python 3.10) with all the corresponding requirements satisfied. Finally, i have cloned the repo (no editing) instead of using the command line tool.
My main concern here is that _acc2 and F1 metrics are almost 1% less than yours. This result is of course statistically significant (the corresponding std's are shown after the comma char). Do you have any good reason why am I getting that much of a difference? Could it be aligned /unaligned, any missing hyperparameters, pytorch version etc (mine is 1.13) ??
Thanks in advance.