search

tomaarsen / attention_sinks

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining
https://huggingface.co/blog/tomaarsen/attention-sinks
Apache License 2.0
650 stars 41 forks source link

Error when importing #12

Closed Caet-pip closed 9 months ago

Caet-pip commented 10 months ago

When running this code to import the model:

from attention_sinks import AutoModel model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1", device_map="auto")

I get this error:

ImportError Traceback (most recent call last) /Users/fawazahamedshaik/Documents/RAG/GRails.ipynb Cell 31 line 1 ----> 1 from attention_sinks import AutoModel 3 model = AutoModel.from_pretrained("meta-llama/Llama-2-7b-hf", device_map="auto")

File /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/attention_sinks/init.py:6 3 from transformers import AutoTokenizer 5 from .attention_sink_kv_cache import AttentionSinkKVCache ----> 6 from .models import ( 7 AutoModel, 8 AutoModelForCausalLM, 9 AutoModelForQuestionAnswering, 10 AutoModelForSequenceClassification, 11 AutoModelForTokenClassification, 12 FalconForCausalLM, 13 FalconForQuestionAnswering, 14 FalconForSequenceClassification, 15 FalconForTokenClassification, 16 FalconModel, 17 FalconPreTrainedModel, 18 GPTNeoXForCausalLM, 19 GPTNeoXForQuestionAnswering, 20 GPTNeoXForSequenceClassification, 21 GPTNeoXForTokenClassification, ... (...) 36 FalconModel, 37 )

ImportError: cannot import name 'MistralConfig' from 'transformers' (/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/init.py)

tomaarsen commented 10 months ago

Hello!

MistralConfig was only added to transformers very recently, in version 4.34.0. You must install this version:

pip install transformers==4.34.0

Hope this helps!