search

tomaarsen / attention_sinks

Extend existing LLMs way beyond the original training length with constant memory usage, without retraining
https://huggingface.co/blog/tomaarsen/attention-sinks
Apache License 2.0
650 stars 41 forks source link

Error when using Falcon #8

Closed helleuch closed 10 months ago

helleuch commented 10 months ago

Hello,

When using attention sink with falcon-7b-instruct, I get the following error:TypeError: FalconForCausalLM.__init__() got an unexpected keyword argument 'attention_sink_size'. I added the attention_sink_size and attention_sink_window_size to the AutoModelForCausalLM.from_pretrained in my code.

But in the provided example, it says it is compatible with it. Am I overlooking something ?

Edit: I tried it with mistal-7b and did not get the error.

tomaarsen commented 10 months ago

Did you use trust_remote_code=True? I don't have this anywhere in the documentation right now, but all models should be loaded without trust_remote_code=True.

helleuch commented 10 months ago

Thank you, I will try that.

helleuch commented 10 months ago

That solved the issue :)