search

davmacario / MDI-LLM

Implementation of Model-Distributed Inference for Large Language Models, built on top of LitGPT
MIT License
3 stars 2 forks source link

[FEAT]: Implement new tokenizer (tested) #3

Closed davmacario closed 7 months ago

davmacario commented 7 months ago

Added custom BPE tokenizer, allowing for larger vocabulary sizes.

Solved some minor bugs with parser and made it more user-friendly.