search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
12.52k stars 856 forks source link

tiktoken fails with Rust 1.60 since memchr 2.6.3 got released #197

Closed jhgoebbert closed 12 months ago

jhgoebbert commented 1 year ago

Hi, thank you for this great package. We are using it in combination with JupyterAI.

Lately memchr got updated to version 2.6.3 which requires rust-version = "1.61": https://github.com/BurntSushi/memchr/blob/2.6.3/Cargo.toml#L14
As memchr is a dependency of bstr ( https://github.com/BurntSushi/bstr/blob/1.0.1/Cargo.toml#L32 ) and bstr a dependency of tiktoken Rust 1.61 is now also required to install tiktoken.

I tried to add 

[patch.crates-io]
memchr = "=2.6.2"

to Cargo.toml but this does not work as it gives me

patch for `memchr` in `https://github.com/rust-lang/crates.io-index` points to the same source, but patches must point to different sources

Is there any other possible work-around to ensure that tiktoken keeps its support for Rust 1.60?

hauntsaninja commented 12 months ago

Since you're already willing to patch Cargo.toml, can you just patch it to use the exact versions of memchr and bstr you desire?

jhgoebbert commented 12 months ago

I am not too familiar with that, but the error message does not lead me to a solution. patches must point to different sources

We have now updated to a newer version of Rust - that is now our "fix".

hauntsaninja commented 12 months ago

Ah I meant changing the dependencies directly instead of the patch.crates-io. Anyway, glad you've got things working! :-)