Added fixed and axial positional embedding option

lucidrains / performer-pytorch

An implementation of Performer, a linear attention-based transformer, in Pytorch

MIT License

1.08k stars 141 forks source link

Added fixed and axial positional embedding option #54

Closed gulnazaki closed 3 years ago

gulnazaki commented 3 years ago

Hello Phil,

this PR adds fixed and axial positional embedding support as discussed in #47. Code is similar to reformer implementation, but the fixed pos_embs are computed once.

I was kinda busy so I looked at it now, have a nice 2021!

lucidrains commented 3 years ago

@gulnazaki thank you! you have a safe and healthy 2021 as well :D