search

richardbaihe / paperreading

NLP papers
MIT License
2 stars 0 forks source link

Arxiv2020|Longformer: The Long-Document Transformer #31

Closed richardbaihe closed 4 years ago

richardbaihe commented 4 years ago

https://github.com/allenai/longformer

image

This paper proposes local+global attention for the transformer.

image

achieves SOTA on WikiHop and TriviaQA after pretraining Roberta with Longformer and finetuned with Longformer.

richardbaihe commented 4 years ago

Results

LM

image

QA

image