Open arseniybelkov opened 1 month ago
Can you remove the chat template and the <image>
token in your prompt and simply just do
chat_completion_from_base64 = client.chat.completions.create(
messages=[{
"role":
"user",
"content": [
{
"type": "text",
"text": "What is the content of this image?",
},
{
"type": "image_url",
"image_url": {
"url": f"data:image/jpeg;base64,{image_base64}"
},
},
],
}],
model=model,
)
Getting the following for a simple "Hi": Hi<|im_end Hi<|im_end <|im_start tαΊ£ You are a helpful assistant.
For prompts with images getting only
@ywang96 I replaced the chat template with the one you suggested. I now get:
Chat completion output:<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
<|im_end>
</|im_end>
<|im_end>
</|im_end>
<|im_end>
</|im_end>
<|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
</|im_end>
I also started using this image since the previous one is no longer available
@arseniybelkov In fact, I don't think PaliGemma is supposed to work with OpenAI API format since it's never instruction fine-tuned and it never came with a chat-template. Can you try this model with LLM
class from vLLM and see if you're still getting the same result?
Sorry for delay, I've been on vacation.
from vllm import LLM
from PIL import Image
import PIL
import requests
from io import BytesIO
response = requests.get("https://upload.wikimedia.org/wikipedia/commons/thumb/1/15/Tour_Saint-Jacques_au_cr%C3%A9puscule.jpg/500px-Tour_Saint-Jacques_au_cr%C3%A9puscule.jpg")
image = Image.open(BytesIO(response.content))
# Refer to the HuggingFace repo for the correct format to use
prompt = "What is the content of this image?"
# Load the image using PIL.Image
llm = LLM(model="google/paligemma-3b-mix-224")
# Single prompt inference
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": image},
})
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
This code results in this output
Processed prompts: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:00<00:00, 22.63it/s, est. speed input: 6028.09 toks/s, output: 67.96 toks/s]
GHFW
@arseniybelkov I don't think the model is supposed to work that way - could you try caption es
as the prompt? (This is the same example given in the documentation from huggingface, which is # Instruct the model to create a caption in Spanish
)
You can refer to the example here
Yes, you're right, I didn't pay attention to it. I changed the prompt to "caption en"
and now it gives me this
In this image there are trees, at Iliad intersectionδΈͺδΊΊ, at the top there
@ywang96 ping just in case
Yes, you're right, I didn't pay attention to it. I changed the prompt to
"caption en"
and now it gives me thisIn this image there are trees, at Iliad intersectionδΈͺδΊΊ, at the top there
It does seem that the generation does start correctly...which precision is this?
I've just set dtype=torch.float32 and got the same result
@arseniybelkov Is it possible if you can send me the image and the prompt so I can repro and compare to transformers
(especially on input embeddings)?
@ywang96 Here is the code and image
from vllm import LLM
from PIL import Image
import PIL
import requests
from io import BytesIO
from transformers import AutoTokenizer
import torch
MODEL_ID = "google/paligemma-3b-mix-224"
image = Image.open("Notre-Dame_de_Paris,_4_October_2017.jpg")
# Refer to the HuggingFace repo for the correct format to use
prompt = "caption en"
# Load the image using PIL.Image
llm = LLM(model=MODEL_ID, tokenizer=MODEL_ID, dtype=torch.float32)
# Single prompt inference
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": image},
})
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
Hey @arseniybelkov I just want to let you know this is still on my list of TODO but I simply had limited bandwidth with other priorities. Sorry for the inconvenience!
Your current environment
π Describe the bug
I run the following docker container
And after it is up & running, I execute the following python code
This results in the following ooutput:
Please help.