Closed YoungloLee closed 10 months ago
@YoungloLee hey Younglo
yes i think you are right
want to check 0.2.6 see if it aligns better? also, do you know if the tanh
output aligns with the reals, and the glu
output aligns with the imaginary? i don't have enough domain expertise to know why one mlp output is bounded between -1
and 1
while the other is not
@lucidrains
According to the paper, tanh activation output is directly fed into the next fully connected layer with glu activation:
MLP: Sequential(RMSNorm, Linear, tanh, Linear, GLU))
I'm not certain, but those two don't seem to be related to the real and imaginary components. It just appears to be the authors' preference for an activation function
@YoungloLee ahh i totally misunderstood, thank you! i'm used to seeing swish for activation functions
ok, 0.2.8 should be accurate to the paper then!
Looks pretty well now and the parameters getting really close to the original paper but I think the RMS Norm they mentioned is still missing
@Psarpei it is taken care of at the end of the attention layers https://github.com/lucidrains/BS-RoFormer/blob/main/bs_roformer/bs_roformer.py#L302
it is standard to do a final normalization at the end of pre-normalized transformer
Aaaaah I see thanks :)
Thanks for your great work. In the BS-RoFormer paper, the authors mention:
However, your MaskEstimator implementation is somewhat different from the paper description.
Here's my implementation (I also checked the number of parameters):