Open luohao123 opened 6 months ago
The image resolution should keep to 384 in the first stage, then you can increase the resolution in the second stage. Notbly, the image size should be divisible by 32 for accommodating ConvNeXt's setting.
From the script in your pretrain, I clearly see you have set input size to 384 while 336 in original clip model.
So here have some issues for me to clarify:
thanks for your reply
384 is the resolution of ConvNeXT. And the resolution of CLIP is still 336 in the first stage. See bellowing codes in multipath_encoder_wapper.py, where we downsample the resolution to 336 for CLIP.
fast_image_size=max(int(self.image_size/32*14),336) y = F.interpolate(x.float(), size=(fast_image_size, fast_image_size), mode='bilinear', align_corners=True).to(dtype=x.dtype)
If you use Siglip-384 with a stride of 16 as the visual backbone, you may need modify self.image_size/32*14
to self.image_size/32*16
. Then, you can directly run our script.
384 is the resolution of ConvNeXT. And the resolution of CLIP is still 336 in the first stage. See bellowing codes in multipath_encoder_wapper.py, where we downsample the resolution to 336 for CLIP.
fast_image_size=max(int(self.image_size/32*14),336) y = F.interpolate(x.float(), size=(fast_image_size, fast_image_size), mode='bilinear', align_corners=True).to(dtype=x.dtype)
If you use Siglip-384 with a stride of 16 as the visual backbone, you may need modifyself.image_size/32*14
toself.image_size/32*16
. Then, you can directly run our script.
I don’t recommend you to increase the resolution in the first stage, which will hurt the final performance.
Got loss 0, any ideas?
am first trained a Projector with input size of 490, and do sft based on it, am direclty interpolate just like in your repo does.