Read 311990 words (17104 unique) from text file. terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped)

glample / fastBPE

Fast BPE

MIT License

656 stars 96 forks source link

Read 311990 words (17104 unique) from text file. terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped) #52

Open dinaalaaahmed opened 2 years ago

dinaalaaahmed commented 2 years ago

/home/dina/code/fastBPE/fast learnbpe 500 /home/dina/CodeGen/data/test_dataset/cpp.sa-cl.tok.shuf.50gb > /home/dina/CodeGen/data/test_dataset/cpp.sa-cl.codes Loading vocabulary from /home/dina/CodeGen/data/test_dataset/cpp.sa-cl.tok.shuf.50gb ... Read 311990 words (17104 unique) from text file. terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Aborted (core dumped)