[LLM][Experiment] Tokenization model using lexical patterns - Githubissues

dishmint / LexicalCases

Extract substrings matching a lexical pattern

https://www.paclets.com/FaizonZaman/LexicalCases

MIT License

2 stars 0 forks source link

[LLM][Experiment] Tokenization model using lexical patterns #302

Closed dishmint closed 6 months ago

dishmint commented 6 months ago

Training a tokenizer on lexical patterns. Does/Can this do anything useful?