Training TinyViT on custom dataset

[abhishekmann]

Can you please share a recipe on transfer learning of TinyViT pretrained on Imagenet 1K/22K to a different dataset including knowledge distillation from a teacher for the same downstream task. Thanks.

[wkcn]

Hi @abhishekmann, thanks for your attention to our work!

The hyper-parameter can refer to moco-v3 for linear probe, cdfsl-benchmark for few-shot learning, mmdetection for object detection.

For transfer learning on image classification, you can refer to our another paper: MiniViT, Section 4.4 Transfer Learning Results.

We only apply knowledge distillation to pretrain TinyViT on ImageNet 22k.

If you want to apply knowledge distillation on downstrem datasets, the teacher model should be fine-tuned firstly, then distill the student model by the same hyper-parameter except that the drop path rate is set to 0. Saving Teacher logits is available.

[Shyamji07]

Hi is there any code available for finetuning the model on custom classes and dataset? Thanks :)

[wkcn]

@Shyamji07 Thanks for your attention to our work!

Sorry that we did not provide the finetuning script on custom dataset. You can refer to our hyper-parameter and the setting in MiniViT Sec 4.4 for downstream classification datasets.

[Shyamji07]

@Shyamji07 Thanks for your attention to our work!

Sorry that we did not provide the finetuning script on custom dataset. You can refer to our hyper-parameter and the setting in MiniViT Sec 4.4 for downstream classification datasets.

Thank you for the response @wkcn Will follow these resources.

[wkcn]

@Shyamji07 TinyViT has been integrated into mmpretrain and timm. You can refer to their fine-tune scripts : )

[Shyamji07]

@Shyamji07 TinyViT has been integrated into mmpretrain and timm. You can refer to their fine-tune scripts : )

Thanks a ton @wkcn :)