LED models give: `IndexError: index out of range in self`

[nicola-decao]

Using a Longformer on more an input with than 1024 tokens doesn't seem to work. But it should as the Longformer was specifically designed to handle up to 16k tokens as input. See the code below.

To reproduce:

from transformers import LEDTokenizer, LEDForConditionalGeneration

tokenizer = LEDTokenizer.from_pretrained("allenai/led-base-16384")
model = LEDForConditionalGeneration.from_pretrained("allenai/led-base-16384")

# this works (tokens < 1024)
model(**tokenizer("hello " * 120, return_tensors="pt"))

# this does not work! (tokens > 1024)
model(**tokenizer("hello " * 1200, return_tensors="pt"))

Versions:

• python 3.8.11
• pytorch 1.11.0.dev20210928+cu111
• transformers 4.11.0

[nicola-decao]

Sorry I realized I should post this in huggingface/transformers