search

allenai / unifiedqa

UnifiedQA: Crossing Format Boundaries With a Single QA System
https://arxiv.org/abs/2005.00700
Apache License 2.0
428 stars 43 forks source link

Multi-Paragraph #10

Closed jiajudu closed 3 years ago

jiajudu commented 3 years ago

How to process documents when their length exceed the maximum number of Bart/T5's input tokens?

danyaljj commented 3 years ago

We haven't studied such cases. I suppose you can split your into smaller chunks and solver them separately.

chen-yifu commented 3 years ago

Hi, can someone please tell me what is the "maximum number of Bart/T5's input tokens"? Thank you in advance!

jiajudu commented 3 years ago

For example, Bart can process 1024 tokens at every turn.