search

knuddelsgmbh / jtokkit

JTokkit is a Java tokenizer library designed for use with OpenAI models.
https://jtokkit.knuddels.de/
MIT License
516 stars 38 forks source link

feat: Implement o200k_base encoding and support gpt-4o #99

Open chatanywhere opened 1 month ago

chatanywhere commented 1 month ago

Implement o200k_base encoding and support gpt-4o

fernando-arellano commented 1 month ago

Hello @tox-p thanks for all the support on maintaining this repo. Is there anything we can do to help you with this PR? we're relying on this library in our project and righ now we're looking to start using gpt4o but it seems the library doesn't have support yet for gpt4 o. please let us know if there is something we can do to help with this.

jcrupi commented 3 weeks ago

Checking in on this approval

mkemmerz commented 2 weeks ago

@tox-p Sorry to annoy you directly but any plan to review this PR?

keyurgolani7878 commented 1 week ago

Hey, @tox-p Can you provide update on when are you planning to review the PR? I need gpt-4o encoding ASAP. Thanks!

Grogdunn commented 1 week ago

meanwhile :zipper_mouth_face: 

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>
keyurgolani7878 commented 1 week ago

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

I can't find this on maven central. Can you help me?

Grogdunn commented 1 week ago

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

I can't find this on maven central. Can you help me?

Strange, I've released this morning and I see in central portal: https://central.sonatype.com/artifact/net.optionfactory/jtokkit

mkemmerz commented 1 week ago

meanwhile 🤐

<dependency>
    <groupId>net.optionfactory</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.1</version>
</dependency>

I can't find this on maven central. Can you help me?

Strange, I've released this morning and I see in central portal: https://central.sonatype.com/artifact/net.optionfactory/jtokkit

Not sure what dependency that is but not the one from this Github repo, this is the correct maven repo: https://mvnrepository.com/artifact/com.knuddels/jtokkit

Grogdunn commented 1 week ago

@mkemmerz of course that dependency has different groupId. It came from a fork with merged support for GPT-4o and GPT-4-turbo models. Nothing "strange", I've need of this functionality for some customers, so I've published myself. Here the fork https://github.com/optionfactory/jtokkit and thanks for the peole who made the PR.

joesh36 commented 1 week ago

Any progress made to merge it to the main repo?