Open dansoutner opened 4 years ago
Hello @dansoutner!
Thanks for the tip. I actually experimented with removing the batch norm at the end and using cosine distance, but it decreased the results so I didn't push it further. Do you keep the batch norm and apply cosine distance?
Used output embedding as they are now in your code, just computed cosdist insted of euclidian. I tred this on two (relatively) small datasets and helps a little, but noticeable (MR 5.43 vs 5.86).
Ah okay, I thought you trained it with cosine distance as well. I'll add the option to use it for both training and evaluation.
Thanks again for the tip!
Small improvement tip: cos distance seems to performs better, when I tried on my data. Worth trying to implement both along euclidian and make more tests.