Spanish test accuracy pretty low (> 10%)

[neelpawarcmu]

Hello authors, I am trying to reproduce these results on the Spanish dataset, using all default parameters. I do understand that without joint loss there should be lower accuracy than with joint loss, but the difference is around 10 to 20 percent. I checked for overfitting and it does not seem to be an issue, plus models are being saved only for best cases of val loss. Do you reckon changing any of the default parameters per se?

Here are results on the test set: F1-Micro: 0.4789 (expected 0.654) F1-Macro: 0.3418 (expected 0.534) JS: 0.3665 (expected 0.481)

Thanks for your time!

[hasanhuz]

Hi there, thank you for your interest in our work.

It's weird, there might be something in the pre-processing pipeline or in the hyper-parameters. It might be useful to print those settings before running the model, just to ensure that the parameters are set as described in the paper.

Hope this helps, Hassan