-
Hey,
I'm trying to add a large number of new tokens (20k) to a pretrained tokenizer, but after adding the new tokens the needed time to tokenize the same data jumps from 1 minute to more than 10 hou…
-
## Environment info
- `transformers` version: 4.16.2 (issue exists on 4.9.2)
- Platform: Linux-4.4.0-210-generic-x86_64-with-glibc2.10
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.1+…
-
## Environment info
- `transformers` version: 4.11.0
- Platform: macOS Big Sur
- Python version: 3.8.5
### Who can help
@LysandreJik
## Information
When using the AlbertTokenizer or A…
-
From the recommendation on issue #5231
I start with a blank model and modify the existing tokenization rules, then save the model to disk and use that as my base model for downstream tasks includin…
-
## ❓ Questions and Help
Thanks for releasing the mbart models! However, we are unable to produce the EN-RO fine-tuned BLEU scores reported in the paper. We get a BLEU score of 26.9, using sacreBLEU…
-
I trained on my customed dataset.
python interpret_prompt.py give the output below:
`Return the top-3 matched words
Size of token embedding: torch.Size([49408, 512])
Size of context: torch.Size([1…
-
Note, 13th of February 2023: Next de facto discussion place until further notice is at #779.
————————————
So today I learned that [GitHub threads max out at 2,500 comments](https://github.com/Da…
-
## How to reproduce the behaviour
The 'recomended way' of upgrading to Spacy3 is '**re-train your models**'. To justify the upgrade one would expect that re-training (using the same data) will result…
-
**Describe the bug**
A clear and concise description of what the bug is.
Not being able to save T5-11B checkpoint using deepspeed
**To Reproduce**
Steps to reproduce the behavior:
```
export…
-
Hi,
Is there a possibility to reproduce results for NER on CPU instead of the default GPU configuration? I am unable to find any resource for this on the repo.
I am using the following command, bu…