-
I am trying to train a custom tokenizer. My use case is related to assembly code, so I want merges to be possible across full instructions (potentially multiple "words"). To do this, I am replacing al…
-
The relationship of BPE, SentecePiece, and byte-fallback needs to be cleared up.
-
Hi,
I'm interested in contributing to implementing the BPE tokenizer.
Since we're using gpt-2 encoding (as shown in the preprocessors), I think we can use the original implementation of `tiktoke…
-
hi, @frankyoujian , I have successfully ran the sherpa-onnx version, but
I got a batch of sentences want to be decode, I may need to use file to be more efficient.
when I try to run ,
1. …
-
Thanks for the nice repo! 🙂
Not sure if it's welcome here given that the goal of this repo is to be for educational purposes but you call the `get_stats` function, which is computationally heavy, `…
-
### What happened?
I ran into this issue when working on a [PR](https://github.com/huggingface/transformers/pull/31844) on HF where I was adding GGUF support for phi3 model.
When using gguf-my-…
-
Could be fun to have a tokenizer like "take all video frames, apply clip, transform into cluster N of 2^17 (what I have in clip retrieval index), apply BPE, return sequence"
Inspired by https://arx…
-
@ArthurZucker I am trying to train a bytepiece tokenizer on my dataset. I have a list of words which I want to be treated as a single token. But when I train it and tokenize, I observe that the token …
-
Hi,
nice project, thanks! I'm just trying to replicate your setup for IWSLT'14. Did you change the variable `BPE_TOKENS` in fairseqs `prepare-iwslt14.sh` to 32k as mentioned in your paper?
Are …
-
Hello,
we have noticed some unexpected behaviors when fine-tuning a llama 3 model on 1 gpu and when fine-tuning the same model on the same data set with 2 gpus in parallel mode. See the attached te…