-
### Description of issue
Package installation fails when building the wheel for tokenizers.
System: macOS M1
Tried: installing and updating Rust and rustup but the error persists. Also attempt…
-
Having a base-class for tokenizers could be useful. It would be like this:
```
class TokenizerBase:
def __call__(self, data: Data) -> Dict:
raise NotImplementedError
```
All toke…
-
### Feature request
Hi @xenova ,
I would like to request the creation of a lightweight node package, `tokenizers.js`. It would be great if developers have flexibility to use `tokenizers.js` instead …
r4ghu updated
1 month ago
-
Related to [this](https://github.com/huggingface/tokenizers/issues/1539).
Currently cache config for BPE tokenizers is set either via `BpeBuilder` or via config in `tokenizer.json` for pretrained t…
-
```
Epoch 019 Step 00270: psnr: 19.72 l_rgb: 0.010838 l_depth: 0.001648 l_trans: 0.000423: 100%|█| 270/270 [00:31
-
### Your question
Error while loading checkpoint:!!! Exception during processing !!! 'tokenizers.AddedToken' object has no attribute 'special'
### Logs
```powershell
got prompt
model weight dtype…
npubg updated
3 weeks ago
-
Hi @Dan-wanna-M!
I wanted to integrate your great work here into mistral.rs and Candle. However, when testing with the `microsoft/Phi-3.5-mini-instruct` model's tokenizer using the below code, I ge…
-
So far I've ported the components I needed to support the models I tested, but there are many more in `transformers` and `tokenizers`. For example:
- https://github.com/huggingface/swift-transforme…
-
# ❓ Questions & Help
## Details
Description: I encountered an error while trying to install transformers4rec. The installation fails when attempting to build the wheel for the tokenizers package…
-
perhaps not for batch inference