Thanks for your great work! And I notice that you utilized a non-linear layer with GELU and a LayerNorm operation and a linear layer called decoder as the voken classification head, which is different from the way mentioned in the paper. In the paper, it is a softmax layer following a linear layer. Did they perform similarly or just cuz I misunderstand it?
Thanks for your great work! And I notice that you utilized a non-linear layer with GELU and a LayerNorm operation and a linear layer called decoder as the voken classification head, which is different from the way mentioned in the paper. In the paper, it is a softmax layer following a linear layer. Did they perform similarly or just cuz I misunderstand it?