Conversion fails for Nous-Hermes-2-SOLAR-10.7B

[epicfilemcnulty]

When trying to convert Nous-Hermes-2-SOLAR-10.7B I get the following error:

python convert.py -i /storage/models/FP16/Nous-Hermes-2-SOLAR-10.7B -o ./tmp -cf /storage/models/EXL2/Nous-Hermes-2-SOLAR-10.7B -hb 8 -b 6.2
Traceback (most recent call last):
  File "/home/epicfile/src/exllamav2/convert.py", line 69, in <module>
    tokenizer = ExLlamaV2Tokenizer(config)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/epicfile/src/exllamav2/exllamav2/tokenizer.py", line 119, in __init__
    self.eos_token = (self.tokenizer.eos_token() or self.extended_id_to_piece.get(self.eos_token_id, None)) or self.tokenizer.id_to_piece(self.eos_token_id)
                                                                                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/epicfile/src/exllamav2/exllamav2/tokenizers/spm.py", line 28, in id_to_piece
    return self.spm.id_to_piece(idx)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/epicfile/.venvs/exl2/lib/python3.11/site-packages/sentencepiece/__init__.py", line 1045, in _batched_func
    return _func(self, arg)
           ^^^^^^^^^^^^^^^^
  File "/home/epicfile/.venvs/exl2/lib/python3.11/site-packages/sentencepiece/__init__.py", line 1038, in _func
    raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.

Seems related to the https://github.com/turboderp/exllamav2/issues/222

I'm using the latest exllamav2, commit 699d127.

[turboderp]

I can't test it right now, but I think you can solve it by creating an added_tokens.json file in the model folder with these contents:

{
  "<|im_end|>": 32000,
  "<|im_start|>": 32001
}

[epicfilemcnulty]

yep, that works, thanks!