Awesome Vision Language Model

Overview

• Awesome Vision Language Model
  • Overview
  • Contrastive Learning
  • PrefixLM
  • Multi-modal Fusing with Cross Attention
  • Masked-Language Modeling / Image-Text Matching
  • No Training

Contrastive Learning

Narrow the distance of latent space between the image and text.

• CLIP 📝Summary
• CoOp 📝Summary
• CoCoOp 📝Summary
• CLOOB
• ALIGN
• DeCLIP
• LiT
• FLAVA 📝Summary

PrefixLM

Unified multi-modal architecture consisting of encoder and decoder. Main tasks are image-conditioned text generation/captioning and VQA.

• SimVLM
• VirTex
• Frozen
• MAPL
• ClipCap
• Flamingo 📝Summary

Multi-modal Fusing with Cross Attention

Fuse visual information into a language model decorder using a cross attention mechanism. Main tasks are image captioning and VQA.

• VisualGPT 📝Summary
• VC-GPT
• Flamingo 📝Summary
• FIBER

Masked-Language Modeling / Image-Text Matching

Combination of MLM and ITM. MLM predicts the masked word by image that annotated more information as bounding box and ITM matches image and caption among many negative captions.

• VisualBERT
• FLAVA 📝Summary
• ViLBERT
• LXMERT
• BridgeTower
• Clinical-BERT 📝Summary

No Training

Without training, just using pretrained model, make two features into one space.

• ASIF
• MaGiC