Autoregressive and Relative Position Embedding Support - Githubissues

allenai / longformer

Longformer: The Long-Document Transformer

https://arxiv.org/abs/2004.05150

Apache License 2.0

2k stars 268 forks source link

Autoregressive and Relative Position Embedding Support #219

Open btyu opened 2 years ago

btyu commented 2 years ago

Hi! Nice work on the Longformer model! I am learning your model and have got some questions:

I guess the code of LongformerSelfAttention has never enabled the autoregressive mode? Since I noticed that the autoregressive parameter is always set False when calling diagonaled_mm_tvm. If it is a bug, could you please fix it asap?
Does this code support relative position embedding, as you mentioned in the paper that rpe is used in the autoregressive LM. If not, could you please release this part of code?

Thank you~