search

TharinduDR / TransQuest

Transformer based translation quality estimation
Apache License 2.0
107 stars 16 forks source link

Smaller Models for MonoTransQuest possible? #37

Closed jbgruenwald closed 2 years ago

jbgruenwald commented 2 years ago

Hi,

I wanted to ask, whether you tried to use smaller models than XLMR-large for MonoTransquest? I tried it with XLMR-base, which works fine and faster than the large version, but I would also want to try even smaller distilled models.

I found out, that in other situations, the multilingual versions of MiniLM are a very good and fast alternative for XLMR. But when I try to train mMiniLM6 or mMiniLM12 for Transquest, it just produces random scores (correlation around 0). So these models seem to be not really helpful. Did you make similar observations and/or have a hint for me?

Btw: I used the XLM-R class for MiniLM, I guess there is no special one needed, since training worked without errors...

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.