Open chethanwiz opened 2 months ago
This is usually caused by conflicting vocabularies in merged models. Would help to know what model this is.
dolphin-2.1-mistral-7B-GPTQ
The model seems to be using the same tokenizer as Mistral, which doesn't define the two ChatML tokens that Dolphin needs. You can try adding an added_tokens.json file to the model directory with this content:
{
"<|im_end|>": 32000,
"<|im_start|>": 32001
}
can someone help me with this error please
Traceback (most recent call last): File "C:\Users\cheth\Music\new chaya\OneReality\OneRealityMemory.py", line 68, in
ExLlamatokenizer = ExLlamaV2Tokenizer(config)
File "C:\Python310\lib\site-packages\exllamav2\tokenizer\tokenizer.py", line 192, in __init
self.eos_token = (self.tokenizer_model.eos_token() or self.extended_id_to_piece.get(self.eos_token_id, None)) or self.tokenizer_model.id_to_piece(self.eos_token_id)
File "C:\Python310\lib\site-packages\exllamav2\tokenizer\spm.py", line 43, in id_to_piece
return self.spm.id_to_piece(idx)
File "C:\Python310\lib\site-packages\sentencepieceinit.py", line 1179, in _batched_func
return _func(self, arg)
File "C:\Python310\lib\site-packages\sentencepieceinit.py", line 1172, in _func
raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.