Can fastBPE use the subword codes file?

glample / fastBPE

Fast BPE

MIT License

656 stars 96 forks source link

Can fastBPE use the subword codes file? #51

Open SefaZeng opened 3 years ago

SefaZeng commented 3 years ago

I use the fastBPE to apply bpe to make it faster. But I want to use the origin codes file learned from subword. And I found a clean difference between the two results. Is it because fastBPE cannot use the codes file learned from subword?