glample / fastBPE

Fast BPE
MIT License
656 stars 96 forks source link

limitVocab #50

Open geert56 opened 3 years ago

geert56 commented 3 years ago

Why is the limitVocab function necessary? And is it correctly implemented? I see that the query string might have a kTokenDelim attached but then the lookup in vocab makes no sense. What is going on here?