search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
129.76k stars 25.78k forks source link

ValueError: LlavaForConditionalGeneration does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please open an issue on GitHub to request support for this architecture: https://github.com/huggingface/transformers/issues/new #28175

Open 1106280506Hx opened 7 months ago

1106280506Hx commented 7 months ago

processor = AutoProcessor.from_pretrained("/gemini/data-2/data/llava") model = AutoModelForPreTraining.from_pretrained("/gemini/data-2/data/llava",load_in_4bit=True,bnb_4bit_compute_dtype=torch.float16,low_cpu_mem_usage=True,attn_implementation="sdpa").to("cuda")

amyeroberts commented 7 months ago

cc @younesbelkada for reference

1106280506Hx commented 7 months ago

can you tell me how to sovle it?

amyeroberts commented 7 months ago

Hi @1106280506Hx,

As the error suggests, SDPA isn't supported in Llava on the version of transformers being run. However, just looking it up, #28107 was merged in a few days ago. You should be able to resolve this by running the dev version of transformers - installing from source: pip install git+https://github.com/huggingface/transformers

1106280506Hx commented 7 months ago

Your answer is meaningful, thanks for your quick reply @amyeroberts