search

lucidrains / x-transformers

A concise but complete full-attention transformer with a set of promising experimental features from various papers
MIT License
4.63k stars 395 forks source link

paper for GLU Mult Bias? #275

Open TimS-ml opened 4 days ago

TimS-ml commented 4 days ago

Hi:

Is GLU's mult_bias originally from this paper? https://arxiv.org/pdf/2202.08906 It mentioned Add Bias and Mult Bias on page 32. I could not find the info in README, and I am not very sure.

Thanks! 😀