Closed dame-cell closed 3 months ago
cc @molbap
Hi @dame-cell , thanks for the issue! Looking into it. I'm calling the processor
with your options and it seems to correctly form labels and the forward method seems also to work:
model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma-3b-pt-224")
processor = AutoProcessor.from_pretrained("google/paligemma-3b-pt-224")
img = [Image.open("/raid/pablo/paligemma/test_docvqa/docvqa_example.png").convert('RGB')] * 3
text = ["Is the withheld tax rate equal to 31%?"] * 3
labels = ["Yes it is absolutely", "Yeah", "Yes it seems so"]
inputs = processor(images=img, text=text, suffix=labels, truncation="only_second", padding=True, return_tensors="pt", max_length=768)
inputs['labels'].shape
#torch.Size([3, 274])
out = model(**inputs)
out.loss
# tensor(5.3981, grad_fn=<NllLossBackward0>)
Can you check the form of the inputs to the model? Also @merveenoyan shared recently a notebook for fine-tuning, might help too https://colab.research.google.com/drive/1x_OEphRK0H97DqqxEyiMewqsTiLD_Xmi?usp=sharing
Hey thank you so much it works now just had to add the suffix
def collate_fn(examples):
texts = [example["question"] for example in examples]
images = [example["image"].convert("RGB") for example in examples]
labels = [example["answer"] for example in examples]
# Tokenize the texts and images
inputs = processor(text=texts, images=images, suffix=labels, return_tensors="pt", padding=True,
truncation="only_second", max_length=MAX_LENGTH,
tokenize_newline_separately=False)
return inputs
once you do this it works perfectly
Great, glad to hear it!
System Info
I was trying to finetune goggle new vision lanaguge model and I keep getting this error
I assumed this problem is with the forward method in Paligemma itself although I'm not sure so I will be tagging @amyeroberts @pacman100 @muellerz
Who can help?
No response
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
here is the colab notebook
This is the error i was getting
Expected behavior
It would start fine-tuning the model and start producing losses