FoundationVision / OmniTokenizer

OmniTokenizer: one model and one weight for image-video joint tokenization.
https://www.wangjunke.info/OmniTokenizer/
MIT License
234 stars 5 forks source link

Native support for multiple resolutions? #16

Open Jason3900 opened 2 months ago

Jason3900 commented 2 months ago

Hey, thanks for the excellent work! One thing that I'm curious about is, does the model only accept square image/video as input? I see there're preprocess functions used on images and videos, either to crop to the center or scale to a equal width and height, such as 256*256. Does it mean if we want to postprocess the video tokens to reconstructed video, we have to scale it back to the original resolution which may use functions like F.interpolate(recon, size=(original_h, original_w), mode='bilinear', align_corners=False). Right?