how to handle the if length of the context exceeds max length (1024)

iedmrc / galois-autocompleter

Galois is an auto code completer for code editors (or any text editor) based on OpenAI GPT-2.

https://usegalois.com

MIT License

95 stars 26 forks source link

how to handle the if length of the context exceeds max length (1024) #1

Closed santhoshkolloju closed 5 years ago

santhoshkolloju commented 5 years ago

How can we handle if the length of the context is greater than 1024 which is max length of GPT-2

iedmrc commented 5 years ago

In GPT-2 you must just limit the input sequence to a fixed length. There is no such a clear way introduced in the original GPT-2 paper, as I understood. There is a great explanation of GPT-2 here by jalammar.

At that point, Transformer-XL introduces new concepts called "Segment-level Recurrence" and "Relative Positional Encodings" . You can check the link here for more details.