Closed bringitup closed 1 year ago
Thanks for the PR. Normally, we only use ggml model so we wouldn't depend on external tokenizers 🤔
This error is raised at build time, from fresh clone you could easily replicate with
cargo b --release
Sure this should and will be quickly fixed by llm-base direct dependency, so this PR will not be necessary in future
I can't replicate this issue using both cargo b --release
and cargo b --release --features {metal, cublas}
... 🤔
I can still replicate it using fresh cria repo
The issue has been already shared in tokenizers
repo : https://github.com/huggingface/tokenizers/issues/1323
Froze tokenizers dep to v0.13.3 to prevent build error on decode signature which change in v0.13.4 Build errors are coming from tokenizers -> llm-base -> cria :