search

Luodian / Otter

🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
https://otter-ntu.github.io/
MIT License
3.52k stars 241 forks source link

A question about the resizing operation #310

Open Richar-Du opened 7 months ago

Richar-Du commented 7 months ago

Thanks for your awesome work in Otter-HD! I have a question about the resizing operation in fuyu. Since fuyu-8b claims that the model can accept images with any resolution as the input, then why does the code first resize all the images into [1080, 1920]? Why not just keep the original size?

Maybe an explanation is to keep the length of input the same, but adding pad tokens after text tokens is also plausible. Could you help answer this question? Thanks in advance :)

gpantaz commented 5 months ago

Hi I was wondering the same question. After some digging on the FuyuImageProcessor I found this line in the resize method

https://github.com/huggingface/transformers/blob/1c31b7aa3bb4e7ef24c77596d2a76f45a770159f/src/transformers/models/fuyu/image_processing_fuyu.py#L299-L303C25

Correct me if I am wrong but I believe that the image is not resized unless it is larger than 1080x1920. Instead, it is padded to these dimensions. This means that say for an image (w, h) this is resized to 1080x1920 with padding values in each dimension. Then to handle variable sized images the processor tries to find the minimum integer that exceeds w (or h) and it is divisible by the patch size (30): w1 or (h1)

https://github.com/huggingface/transformers/blob/1c31b7aa3bb4e7ef24c77596d2a76f45a770159f/src/transformers/models/fuyu/image_processing_fuyu.py#L632C1-L644C65

Since the image has been padded to 1080x1920, it is safe to take w1 cols and h1 rows from the padded tensor.

However, this means that for all benchmarks that do not have images larger than 1080x1920 the performance of the model should be exactly the same in 1080x1920 and the "variable size" scenario? :shrug: