DeiT: Data-efficient Image Transformers

[sctrueew]

Hi @AlexeyAB,

Can we have this DeiT?

Thanks

[AlexeyAB]

@zpmmehrdad I think the most users can't use good Deit models, because they are private and it require significant hardware to: train+fine-tune+distill it.

• DeiT is only faster 2x times than EfficientNet with the same accuracy, only by using Fine-tuning+Distillation and only for Classification (currently).
• While Scaled-YOLOv4 is faster than EfficientDet 2.3x - 3.7x times without Distillation and for Detector.

---

It seems in public there is only model without Fine-tuning-384x384 and without Distillation with 81.8% Top1: https://github.com/facebookresearch/deit#model-zoo

---

In the paper, there are models:

• with Fine-tuning-384x384 and without
• with Distillation and without

---

There are used data-efficient image Transformers & Distillation:

• Teahcer RegNetX-16GF - 82.9 Top1 https://github.com/rwightman/pytorch-image-models/blob/2ed8f247154870be7acc1908fde0a7f457f67456/timm/models/regnet.py#L396-L399
• Student pre-train - 83.1% Top1
• DeiT-B ↑384 - 84.2% Top1
• 500 epochs, train224 + finetune↑384, class+distillation

[sctrueew]

@AlexeyAB Hi,

Thanks for the explanation.