BPE clarifications - Githubissues

elte-nlp / elte-nlp-course

NLP & FM Lecture Slides

Creative Commons Attribution 4.0 International

30 stars 2 forks source link

Closed nata108 closed 1 month ago

nata108 commented 8 months ago

The relationship of BPE, SentecePiece, and byte-fallback needs to be cleared up.

nata108 commented 3 months ago

add extra fixes (SentencePiece as a library/algorithm, whitespace splitting, numeral splitting, comments on greedy inference/merge order)