Closed IlyasMoutawwakil closed 4 months ago
cc @younesbelkada and @fxmarty, if they use static cache then that is expected. I might fix it in #27931
cc @younesbelkada 🤗
was this fixed? i just ran into the same error here
Thanks everyone ! I managed to repro and the fix should be : https://github.com/casper-hansen/AutoAWQ/pull/401 cc @casper-hansen
Note if you run a dummy forward pass before you need to explicitly pass use_cache=False
:
from transformers import AutoModelForCausalLM, AwqConfig, AutoTokenizer
awq_config = AwqConfig(do_fuse=True, fuse_max_seq_len=512)
model = AutoModelForCausalLM.from_pretrained(
"casperhansen/tinyllama-1b-awq",
quantization_config=awq_config,
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("casperhansen/tinyllama-1b-awq")
input_ids = tokenizer("Hello, my dog is cute", return_tensors="pt").input_ids.to("cuda")
model.forward(input_ids, use_cache=False)
model.generate(input_ids, max_new_tokens=100)
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Closing as fixed on the latest autoawq release (see message above), let me know if this issue is still relevant
System Info
transformers
version: 4.36.2Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
code works if only generate is called but not if a forward pass precedes it. looking at the traceback:
the problems seems to be related to the sdpa integration