-
Hello @LostXine, when running llara on multiple GPUs, I encountered the following error:
```
Exception has occurred: RuntimeError
CUDA error: device-side assert triggered
CUDA kernel errors migh…
-
When users:
- Opt-out existing `bert` and `simple` tokenizers.
- Are not community contributors to add new tokenizer as a functionality to SadedeGel
They should be able to feed their tokenized t…
-
# Bug Report
Iam referring to [https://github.com/microsoft/onnxruntime-inference-examples/tree/main/quantization/language_model/llama/smooth_quant](https://github.com/microsoft/onnxruntime-inference…
-
Hi there!
Galactica [tokenizer](https://huggingface.co/facebook/galactica-120b/blob/main/tokenizer_config.json)'s `eos_token_id` is not set but it's set in its model config. We account for tokenize…
-
I would like to rewrite the tokenizer in rust so that I do not have to rely on external dependencies.
It will be a separate crate that could be used by itself.
It will be written in rust.
It will…
-
Hey Mišo
I spent a lot of time on text rank and while digging deeper into Sumy I want to ask you a few clarifying questions about some of the choices you made: This is all for English language.
…
-
### System Info
```shell
Transformers + Tokenizers
```
### Who can help?
This Issue is a summary of multiple problems that we are currently encountering with Tokenizers. To solve them we'll need a…
-
请问下,我加载bge-rerank模型的tokenizer,下面的代码解析出来的如下:
```
query='中国人你好'
title='你好中国人'
res=tokenizer.encode_plus(
query,
title,
add_special_tokens=True,
…
-
```
Traceback (most recent call last):
File "", line 1, in
File "/home/mmior/.local/share/virtualenvs/json-descriptions-cLP9Kwl8/lib/python3.10/site-packages/litgpt/tokenizer.py", line 39, in …
-
I want to use custom tokenizer and encoder trained from huggingface tokenizer.
After training the huggingface tokenizer, I got a json containing vocas.
However, I don't know how to feed this cu…