Mistral-7b Error: AttributeError: 'str' object has no attribute 'tolist'

[JannikSchneider12]

Hey,

I wanted to test the Mistral-7b model and tried to run the following code:

from ctransformers import AutoModelForCausalLM, AutoConfig, Config conf = AutoConfig(Config(temperature=0.8, repetition_penalty=1.1, batch_size=52, max_new_tokens=1024, context_length=2048))

llm = AutoModelForCausalLM.from_pretrained("mistral-7b-instruct-v0.1.Q5_K_S.gguf", hf=True, model_type="mistral", config = conf)

print(llm("AI is going to"))

But I am getting this error:

---

AttributeError Traceback (most recent call last) c:\Users\DELL-XPS\Documents\LLMs\Mistral-7b\Mistral-7b.ipynb Cell 19 line 1 ----> 1 print(llm("AI is going to"))

File c:\Users\DELL-XPS\anaconda3\envs\mistral-7b\Lib\site-packages\torch\nn\modules\module.py:1518, in Module._wrapped_call_impl(self, *args, kwargs) 1516 return self._compiled_call_impl(*args, *kwargs) # type: ignore[misc] 1517 else: -> 1518 return self._call_impl(args, kwargs)

File c:\Users\DELL-XPS\anaconda3\envs\mistral-7b\Lib\site-packages\torch\nn\modules\module.py:1527, in Module._call_impl(self, *args, *kwargs) 1522 # If we don't have any hooks, we want to skip the rest of the logic in 1523 # this function, and just call forward. 1524 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks 1525 or _global_backward_pre_hooks or _global_backward_hooks 1526 or _global_forward_hooks or _global_forward_pre_hooks): -> 1527 return forward_call(args, **kwargs) 1529 try: 1530 result = None

File c:\Users\DELL-XPS\anaconda3\envs\mistral-7b\Lib\site-packages\ctransformers\transformers.py:72, in CTransformersModel.forward(self, input_ids, return_dict, **kwargs) 70 logits = [] 71 for tokens in input_ids: ---> 72 tokens = tokens.tolist() 73 tokens = llm.prepare_inputs_for_generation(tokens) 74 llm.eval(tokens)

AttributeError: 'str' object has no attribute 'tolist'

Can someone help me?

[jsjolund]

If you remove hf=True it should work.