-
求助,大佬为啥明明文件夹里有bert_tokenizer.py,但是却一直说检测不到
!!! Exception during processing!!! No module named 'scripts.bert_tokenizer'
Traceback (most recent call last):
File "E:\SD\SDcomfyui\Blender_ComfyUI\Com…
-
I am using "google/gemma-2b-it" model from HuggingFace. I realized there are 99 unused tokens (\ ,\,\...) in first 106 token ids. Does anyone know their purpose? Just wondering.
-
nice work! starred already.
sorry for asking, why replacing the bos_token with empty string?
sample['positive'] = tokenizer.apply_chat_template(
sample['chosen'], tokenize=False, …
-
unable load Tokenizer using AutoTokenizer.from_pretrained()
errors:
tokenizer = AutoTokenizer.from_pretrained(model_id)
File "/home/ubuntu/venv/lib/python3.10/site-packages/transformers/mod…
-
I am writing to ask for your help with a problem I am having with the tokenizer. I have been trying to solve it for a while now, but I have been unsuccessful.
However, I am having trouble with : Trac…
-
When I run with gpt2 models, all its ok! But when I run with anyone those models exists ridger/MMfreeLM-370M, MMfreeLM-1.3B or MMfreeLM-2.7 this error occur.Why? Can anyone help me?
Error: LLVM ERR…
-
When running the vLLM server for Functionary v2.5 small, the vLLM throws an error because it does not support Functionary tokenizer. I' reverted back to v2.4 for now, but thought I should bring this i…
-
Hello folks, a lot of awesome tools here :) I was going through the tokenizer notebook to understand how the template has changed with tool calling and I wasn't able to run it. It looks like one of th…
-
-
Have you been able to get good results with the tokenization? I've been using a regex like yours to tokenize some texts for my decoder transformer, and the vocabulary size seems to blow up! I think it…