-
**Is your feature request related to a problem? Please describe.**
When using the builtin tokenizer of INCEpTION, it sometimes does errors. It would be nice if the tokenization can be edited.
**De…
-
All the notebooks are giving a lot of errors. `The` PPOTrainer class also has a lot of other arguments that are required to be passed. for example, 'processing_class' instead of 'tokenizer', 'policy'…
-
### Your current environment
The output of `python collect_env.py`
```text
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N…
-
The user should be able to customize the background color, font face, and font size from the config file.
The config file may be named ".aioh" but it will requires some tokenization, or we could simp…
-
### What feature are you requesting?
The ability to create custom stop word removal configurations, so that not all or only certain stop words are removed during tokenization.
### Why are you reques…
-
### System Info
TypeScript 5.5.4
transformers.js 3.0.2
Node.js v20.170
### Environment/Platform
- [X] Website/web-app
- [ ] Browser extension
- [X] Server-side (e.g., Node.js, Deno, Bun)
- [ ] De…
-
FYI
Changes in [transformers tokenizer](https://github.com/huggingface/transformers/issues/31884) gives deprecation warning.
>
>/xxx/dltranslate/lib/python3.12/site-packages/transformers/tokeni…
-
I get this error:
```
Traceback (most recent call last):
File "/home/denis/Documents/ai/unsloth/llama3-chat-template.py", line 20, in
model, tokenizer = FastLanguageModel.from_pretrained(…
-
This is the future warning we are currently reciving:
transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. T…
-
### Your question
Error while loading checkpoint:!!! Exception during processing !!! 'tokenizers.AddedToken' object has no attribute 'special'
### Logs
```powershell
got prompt
model weight dtype…
npubg updated
2 weeks ago