search

allenai / longformer

Longformer: The Long-Document Transformer
https://arxiv.org/abs/2004.05150
Apache License 2.0
2.05k stars 276 forks source link

Longformer for autoregression #212

Open siddagra opened 3 years ago

siddagra commented 3 years ago

How do I use your longformer for autoregressive tasks? The paper says that you have coded longformer for autoregressive tasks but then the GitHub repo is saying that dilated window and autoregression is not supported? Does Lonformer have the capability for autoregression, and if so, how would you recommend I go about training the model (from scratch preferred) for autoregressive tasks. Any guidance is most appreciated. Thank you.