search

lucidrains / g-mlp-pytorch

Implementation of gMLP, an all-MLP replacement for Transformers, in Pytorch
MIT License
417 stars 58 forks source link

Potentially missing the high way pass #8

Open Vincent-Li-9701 opened 3 years ago

Vincent-Li-9701 commented 3 years ago

Hello,

Maybe I missed it, but would you mind pointing out where the high way pass of the gMLP block is in the code? Based on the paper, there is a high way path (addition) between the input and the output. I couldn't find it in the gMLPBlock code.

Thank you

chazzmoney commented 2 years ago

Implemented here: https://github.com/lucidrains/g-mlp-pytorch/blob/main/g_mlp_pytorch/g_mlp_pytorch.py#L39-L45 Used in gMLP: https://github.com/lucidrains/g-mlp-pytorch/blob/main/g_mlp_pytorch/g_mlp_pytorch.py#L233 Used in gMLPVision: https://github.com/lucidrains/g-mlp-pytorch/blob/main/g_mlp_pytorch/g_mlp_pytorch.py#L278