Make tokenizer 100% API compatible with the Huggingface tokenizer - Githubissues

OpenBioML / protein-lm-scaling

Other

54 stars 15 forks source link

Make tokenizer 100% API compatible with the Huggingface tokenizer #47

Open justin-barton opened 9 months ago

justin-barton commented 9 months ago

Make the tokenizer 100% API compatible with the Huggingface tokenizer as proposed by @jamaliki in https://github.com/OpenBioML/protein-lm-scaling/issues/12#issuecomment-1682513840

Some benefits of this change are:

multiprocessing for distributed tokenization
batch_decode methods
attention_masks and special_tokens_masks
simplifies dependencies (doesn't require rust_trie)
enables easy distribution on Huggingface Hub later

justin-barton commented 9 months ago

/take