search

yzd-v / cls_KD

'NKD and USKD' (ICCV 2023) and 'ViTKD' (CVPRW 2024)
Apache License 2.0
217 stars 18 forks source link

Some question about ViTKD #5

Closed peiyingxin closed 2 years ago

peiyingxin commented 2 years ago

Hi, thanks for sharing your great work! I have some question about your work:

  1. where are you get your deit3-base model? Offical model is 85.7 top1 accuracy in ImageNet-1K, paper deit3-base model is 85.48, in addition, official model state_dict is not same with your defined deit3 model state_dict, so you had modified it?
  2. I had used vit-base model from mmcls with 85.43 top1 accuracy distill deit-small from scratch. only got 80.04 top1 accuracy, witch below baseline 80.69, deit3-base model struct is same with vit-base, I'm confused why got this result?

hoping for your reply thank you.

yzd-v commented 2 years ago

1.The weight is transferred from official weight. Beacuse of the environment, the accuracy maybe a little bit different.

  1. ViT-Base is trained with 384x384. While DeiT-S is trained with 224x224. Please be more careful.