It stuck when encoding large amount of text.

latitudegames / GPT-3-Encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3

MIT License

716 stars 196 forks source link

It stuck when encoding large amount of text. #43

Open Alokkumar8 opened 12 months ago

Alokkumar8 commented 12 months ago

I tried to encode text of large PDF of size 11MB. The text must be more than 100k tokens in size. But the gpt-3-encoder failed to process this amount of text data without throwing any error. The program is stuck forever on this line const encoded = encode(textOfDocument);

How to solve?