-
Previous and current works on LLM integration to DISCO relies on pre-tokenized datasets and doesn't account for token decoding after inference.
Full tokenizer support would allow:
- [x] To input n…
-
**Motivation:**
Right now, the chunking uses a greedy algorithm. The following would output the following chunks:
```rust
let text = "Sentence 1. Sentence 2. Sentence 3. Sentence 4.";
splitter.chunk…
-
We have developped a [logger](https://docs.lavague.ai/en/latest/docs/learn/local-log/) to make it easier to track each run.
While we keep the full prompt, inference time and so on, we do not have i…
-
# Problem
Installing latest version of langchain will now break due to latest version of orjson (3.10.7)
# Temporary Solution
Lock the version: orjson==3.10.6 and it works
# Logs
```bash
!pi…
-
I have a win7 32-bit virtual machine that needs to use tiktoken, but it is not available in pip.
-
gpt-4o uses a new tokenizer `o200k_base`, already available in the original tiktoken library
-
I just tried to get this package to work, but it fails on the first attempt
```
julia > using Whisper, Flux
julia > Whisper.transcribe(
"/Users/frkkan96/Desktop/kaa_yw_pb_16000.flac", "/Users…
-
**Describe the issue:**
I'm running into a problem right now when dealing with large code repositories. When I'm dealing with small code repositories, autopilot works great. But when I'm dealing wi…
LZING updated
2 months ago
-
### Duplicates
- [X] I have searched the existing issues
### Summary 💡
was wondering when this would be supported or how i might add it myself if it's just a text value in a collection. the applic…
-
It’s spring. And new encodings came out with new model.
https://platform.openai.com/docs/models
https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74cdb96406c7f3d9add0ae2f8