I maybe noticed an inconsistency between the information in the paper and in the code. In the Alike paper and in the Aliked paper it is specified that the trel temperature is equal to 1.0 In the training code it is 0.1.
This difference could be explained by a modification of the training code afterwards but in practice the value of 0.1 seems more "plausible" than the value of 1.
Intuitively, it seems logical that the temperature should be low to allow the model to converge (a temperature of 1 seems too brutal).
This hypothesis is confirmed by experience: with trel=1 the loss does not converge. But with a value of 0.1 there is convergence.
Do you have any information to share on this subject?
Thank you in advance for these clarifications.
Hello !
Thanks again for this great work!
I maybe noticed an inconsistency between the information in the paper and in the code. In the Alike paper and in the Aliked paper it is specified that the
treltemperature is equal to 1.0 In the training code it is 0.1.This difference could be explained by a modification of the training code afterwards but in practice the value of 0.1 seems more "plausible" than the value of 1.
Intuitively, it seems logical that the temperature should be low to allow the model to converge (a temperature of 1 seems too brutal).
This hypothesis is confirmed by experience: with
trel=1the loss does not converge. But with a value of 0.1 there is convergence.Do you have any information to share on this subject? Thank you in advance for these clarifications.