Open jfpuget opened 6 hours ago
In https://github.com/lucidrains/nGPT-pytorch/blob/main/nGPT_pytorch/nGPT.py#L350 token_embed is normalized along last dimension when it should be normalized along the first dimension.
hey Jean-Francois, it is actually defaulted to first dimension here
In https://github.com/lucidrains/nGPT-pytorch/blob/main/nGPT_pytorch/nGPT.py#L350 token_embed is normalized along last dimension when it should be normalized along the first dimension.