InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

[runhani]

https://arxiv.org/pdf/2404.06512.pdf https://github.com/InternLM/InternLM-XComposer

preview

• 건강검진 다녀왔는데 이 논문에 대한 토론이 격정적(?)으로 일어나서 issue에 등록함
• 자세한 분석 좀 부탁드립니다!

[hjeun]

1. InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition(arxiv)

• Pre-training --> Multitask Training --> Instruction-tuning
• EVA-CLIP 224x224
• Perceive Sampler
  • attentive pooling
  • 257 --> 64
  • BERT-base w/ cross-attention
• InternLM-Chat-7B
• Pre-training
  • 1.1B Images
  • Training Perceive Sampler and LLM
• SFT
  • Multi-task Training: scene understanding, location understanding OCR, vqa, multi-round conv.
  • LoRA LLM, rank and alpha 256
  • Instruction Tuning: pure-text and LLaVA-150K, LRV

2. Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Models(arxiv)

• OpenAI ViT-L-336
• InternLM2-7B-Chat
• Partial LoRA
• Pre-training
  • Training visual encoder and partial lora
• SFT
  • jointly fine-tune visual encoder, LLM, and partial lora

3. A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD(arxiv)

• CLIP ViT-L-14-336
• InterLM2-7B
• Partial LoRA
• Simple Merge Operation: Channel-wise concat nearby 4 tokens and MLP
• add \n token at the end of each row
• Dynamic Image Partition-
• pre-training
  • Freeze LLM, Training VE and Partial LoRA
  • batch size 4096
  • 2 eps
  • lr 2x10^-4
  • warm up 1%
  • HD-25 (336x5)
• 4KHD SFT
  • batch size 2048
  • 3500 steps
  • lr 5x10^-5
  • LLM lr scale factor 0.2