search

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models
https://docs.rs/llm/latest/llm/
Apache License 2.0
6.06k stars 350 forks source link

Fix embedded tokenizer #444

Closed skirodev closed 6 months ago

skirodev commented 6 months ago

Related #434

Fix embedded tokenizer and enable it.

philpax commented 6 months ago

Thanks for this! I made a few minor changes to clean up vestigial code, but nothing major. I've tested this locally and it all seems to work - much appreciated :slightly_smiling_face:

skirodev commented 6 months ago

No problem at all! I am delighted to contribute to this excellent project. It's great to hear that the changes I made were able to clean up the unnecessary code.