Closed xenova closed 3 weeks ago
One way to get it running is to specify the vision inputs:
- new_inputs = processor(decoded, return_tensors="pt", padding=True)
+ new_inputs = processor(decoded, images=image, return_tensors="pt", padding=True)
(but it's still odd why llama works and phi doesn't without the vision inputs) 👀
Here's a full example of it working (but inefficiently):
import requests
from PIL import Image
import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration
model_id = "xtuner/llava-phi-3-mini-hf"
prompt = "<|user|>\n<image>\nWhat are these?<|end|>\n<|assistant|>\n"
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
model = LlavaForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
).to(0)
processor = AutoProcessor.from_pretrained(model_id)
raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)
# Generate w/o past_key_values
output = model.generate(
**inputs,
max_new_tokens=3, # Stop early so we can test out continuing it later
do_sample=False,
return_dict_in_generate=True,
)
print(processor.decode(output.sequences[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))
# outputs "These are two"
# Generate w/ past_key_values
continued_ids = model.generate(
output.sequences,
pixel_values=inputs.pixel_values,
attention_mask=torch.ones_like(output.sequences),
do_sample=False,
past_key_values=output['past_key_values'],
max_new_tokens=20,
)
print(processor.decode(continued_ids[0][inputs.input_ids.shape[-1]:], skip_special_tokens=False))
# outputs "These are two cats sleeping on a pink couch.<|end|><|end|><|end|><|end|><|endoftext|>"
As I understand it, the vision encoder is still run and the inputs are still merged, even though they will be cropped out later and the past_key_values will be used.
Hey!
Llama models were the first ones to get new features like StaticCache or a method to _update_causal_mask
based on the input tensor. That is why the code fails for Phi and not Llama.
But the true error lies in the way we handle vision language models. Right now we expand the input embedding inside modeling file by concatenating image embeddings with text embeddings. When we try to continue generate, the past key values
hold tensors of much larger seq length than the input_text
we feed here, and therefore shapes of keys/values do not match the shape of the attention mask.
continued_ids = model.generate(
output.sequences,
do_sample=False,
past_key_values=output['past_key_values'],
max_new_tokens=20,
)
One way to get it running is to specify the vision inputs:
Yes, this way we can trigger expansion of inputs by concatenating image embeddings, so that the final shapes match with past cache.
Yes, the best solution is to move inputs expansion by dummy values into the processors. I believe @amyeroberts is working on it.
Thanks for the great explanation @zucchini-nlp! Indeed the differences between phi and llama confused me a bit (my use case is support for this in Transformers.js), so I've now taken this into account in my implementation.
Although, this is still problematic for when images are passed. I'll continue looking into this.
@zucchini-nlp this means updating LLMs to the new cache format will remove this error, correct?
@gante no, it will not as past key values still will not match the size of the input sequence. The code will fail on identifying the correct start and end for cache_positions
(@zucchini-nlp since this is VLM related, I'm going to leave you in charge of fixing this issue :) )
Oke, the draft PR is ready, let's link it here for visibility
(Will be fixed by #30962)
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
bump
System Info
transformers
version: 4.38.2Who can help?
@gante (generate) @susnato (phi implementation) @younesbelkada (llava implementation)
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Following the multi-round conversation tutorial from here, I put together this minimal reproduction to show how switching Llava to use a Phi text model (instead of e.g., llama) results in an error when reusing past key values.
Running:
results in this error
Expected behavior
If you try with a llama model (e.g., here; see comments) it works correctly.