GPT2 minicons surprisal: IndexError: index out of range in self

[joyce9936]

System Info

I am trying to calculating surprisal value by feeding in a txt file with about 5000 sentences. But there is a error message I encounter: IndexError: index out of range in self Can anyone help with this issue?

Thank you!

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

Here is the code:

Here is the error message:

Expected behavior

I would like to have the surprisal value for each word for the whole text file.

[ArthurZucker]

Sorry but this seems unrelated to transformers and rather related to the scorer library.