ImageNet SOTA (BASIC-L+Lion finetune)

[zc12345]

ImageNet SOTA

inspiration:

• 学习一下最新的ImageNet sota用了哪些技术

BASIC

• paper: Combined Scaling for Zero-shot Transfer Learning
• method
  • 方法本身是类似CLIP和ALIGN的image-text对比学习
  • contribution
    • 更多数据：6.6B脏数据image-text pair(=4xALIGN/16xCLIP)
    • 更大模型：3B参数(=3.75x参数，8xFLOPS)
    • 更大batch siz：65535(=4xALIGN/2xCLIP)
    • insight：在image-text对比学习中更大batch size带来更好性能

Model soups

• paper: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• code: model-soups
• method
  • 对不同超参下训出来的model
    • uniform soup: 将模型超参累加之后求平均
      • 
    • greedy soup: 从best模型开始，每次滑动平均加入新模型参数，如果acc变好就更新参数
      • 

Lion Optimizer

• paper: Symbolic Discovery of Optimization Algorithms
• code:
  • timm.optim.lion
  • lion
• method
  • 相较于之前的AdamW需要存储一阶和二阶动量，Lion只需要存一阶动量
  • 
  • pseudo code

    def train(weight, gradient, momentum, lr): 
    update = interp(gradient, momentum, β1) 
    update = sign(update) 
    momentum = interp(gradient, momentum, β2) 
    weight_decay = weight * λ 
    update = update + weight_decay 
    update = update * lr 
    return update, momentum

• 使用Lion optimizer进行finetune之后的BASIC-L模型，相较于之前使用Adafactor优化器的ImageNet 90.9%->91.1%