[untested] Add gpt-4o support (o200k_base)

IAPark / tiktoken_ruby

Unofficial ruby binding for tiktoken by way of rust

MIT License

118 stars 26 forks source link

[untested] Add gpt-4o support (o200k_base) #35

Closed marckohlbrugge closed 4 months ago

marckohlbrugge commented 4 months ago

GPT-4o uses o200k_base which this gem currently does not support.

I tried adding it by looking at the Python library. I have NOT tested this code. I'm unfamiliar with Rust and wasn't able to compile it.

Just putting this PR out here to hopefully get the ball rolling on supporting this.

ScotterC commented 4 months ago

I can get CI running but sadly also unfamiliar with Rust. I'm seeing "cannot find function o200k_base in crate tiktoken_rs" which makes me think a compilation of the rust code needs to be included.

@IAPark would know quickly

IAPark commented 4 months ago

I think the upstream library needs updated. I'll look into it tomorrow morning

IAPark commented 4 months ago

Looks like @CaffeeLake beat me to it, closing this now that their PR is merged