search

microsoft / Cream

This is a collection of our NAS and Vision Transformer work.
MIT License
1.66k stars 225 forks source link

Please open source the teacher logits #127

Closed sanyalsunny111 closed 1 year ago

sanyalsunny111 commented 2 years ago

Dear Authors,

Very impressive work. For reproducibility purposes could you please share the teacher logits files for all the teachers shown in this paper?

wkcn commented 1 year ago

Thanks for your attention to our work!

Sorry for the late reply.

We have uploaded the teacher logits files, which were generated by CLIP-ViT-Large/14-22k on ImageNet-22k (Ref. the tutorial).

You can download them by azcopy.

azcopy cp "https://tracking2019.blob.core.windows.net/tinyvit-teacher-logits/clip_224_22k_top100_logits_bs128/?sv=2020-04-08&st=2022-10-26T02%3A29%3A41Z&se=2022-11-09T02%3A29%3A00Z&sr=c&sp=rl&sig=4bkaC8KkG9n2CUv%2BwbE6aVOi%2Fp3yZQnt3vQeIXW6Bc8%3D" ./ --recursive

There are 1440 files (90 epochs * 8 ranks * (key + value files)) about 499 GB (18GB for keys, 481GB for values).

You can download the logits in the specific epoch. For example, download the logits in the epoch 0:

azcopy cp "https://tracking2019.blob.core.windows.net/tinyvit-teacher-logits/clip_224_22k_top100_logits_bs128/logits_top100_epoch0/?sv=2020-04-08&st=2022-10-26T02%3A29%3A41Z&se=2022-11-09T02%3A29%3A00Z&sr=c&sp=rl&sig=4bkaC8KkG9n2CUv%2BwbE6aVOi%2Fp3yZQnt3vQeIXW6Bc8%3D" ./ --recursive

The expiration time is Nov. 09, 2022. We can extend the date if necessary : ).

wkcn commented 1 year ago

Close it. Feel free to re-open it if necessary for extending the date : )

HaoWuSR commented 1 year ago

Could you please extend the date？ Thanks a lot !

wkcn commented 1 year ago

Could you please extend the date？ Thanks a lot !

Hi @HaoWuSR , the date has been extended.

sanyalsunny111 commented 1 year ago

Hello Sir could you also share the logits of teacher based on datasets such as imagenet-1K and any other dataset?

wkcn commented 1 year ago

Hi @sanyalsunny111 , sorry that I did not prepare the teacher logits on ImageNet-1k for this codebase.

You can follow the tutorial to save the sparse logits.

sanyalsunny111 commented 1 year ago

Thank you for responding @wkcn Could you please provide a running script that could be used to save sparse logits of CLIP-ViT-Large/14-22k with imagenet-1k? I am asking cause I see save_logits.py can use both imagenet-22k and imagenet-1k. Any feedback on this?

wkcn commented 1 year ago

Hi @sanyalsunny111, the following command is used to save sparse logits of CLIP-ViT-Large/14-22k on ImageNet-1k.

python -m torch.distributed.launch --nproc_per_node 8 save_logits.py --cfg configs/teacher/clip_vit_large_patch14_22k.yaml --data-path ./ImageNet --batch-size 128 --eval --resume checkpoints/clip_vit_large_patch14_22k.pth --opts DISTILL.TEACHER_LOGITS_PATH ./teacher_logits_1k AUG.MIXUP 0.8 AUG.CUTMIX 1.0 DATA.DATASET imagenet TRAIN.EPOCHS 300 DISTILL.LOGITS_TOPK 10

It needs the extra config options AUG.MIXUP 0.8 AUG.CUTMIX 1.0 DATA.DATASET imagenet TRAIN.EPOCHS 300 DISTILL.LOGITS_TOPK 10 to save the 1k logits.

However, I recommend Swin-base/large 22kto1k or BEiT-base/large 22kto1k as the 1k teacher model, since CLIP-ViT-Large/14-22k is not finetuned on ImageNet-1k.

sanyalsunny111 commented 1 year ago

Hi @wkcn Thank you very much for earlier responses. I have saved the teacher logits on teacher CLIP-ViT-Large/14-22k using ImageNet-1k. Could you please provide a pre-training script (with distillation) with saved teacher logits on ImageNet-1k.

wkcn commented 1 year ago

Hi @sanyalsunny111, you can try this script:

python -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/1k/tiny_vit_21m.yaml --data-path ./ImageNet --batch-size 128  --opts DISTILL.TEACHER_LOGITS_PATH ./teacher_logits_1k AUG.MIXUP 0.8 AUG.CUTMIX 1.0 DATA.DATASET imagenet TRAIN.EPOCHS 300 DISTILL.LOGITS_TOPK 10
sanyalsunny111 commented 1 year ago

Just a quick question where the best checkpoints and training logs will be saved?

wkcn commented 1 year ago

Hi @sanyalsunny111 , The checkpoint and log will be saved in the folder ./output by default. You can change it by adding the command --output xxxx when running the script.

sanyalsunny111 commented 1 year ago

Hey @wkcn could you please share your code for the Linear probe with various datasets as shown in Table 5 of the paper?

wkcn commented 1 year ago

Hi @sanyalsunny111 , please refer to moco-v3 and Cross-Domain Few-Shot Learning (CD-FSL) Benchmark.

sanyalsunny111 commented 1 year ago

Hi @wkcn, I have used Linear Probe and a few shot codes from moco-v3 and CD-FSL, but it's not matching your results on Table 5. Is it possible for you to share your codes for the Linear probing and Few shot?

BrendanCReidy commented 1 year ago

Could you please extend the date？ Thanks a lot !

Hi @HaoWuSR , the date has been extended.

Hi @wkcn can you please extend the date?

wkcn commented 1 year ago

Hi @BrendanCReidy, thanks for your attention to our work! The new link: 

azcopy cp "https://tracking2019.blob.core.windows.net/tinyvit-teacher-logits/clip_224_22k_top100_logits_bs128/?sv=2020-04-08&st=2023-01-26T03%3A00%3A21Z&se=2023-02-09T03%3A00%3A00Z&sr=c&sp=rl&sig=lVTekKcg9SAvzF6AmyGJTIv%2FXNx42XLLqSYSfpxmWLY%3D" ./ --recursive
Andrewymd commented 1 year ago

Hi @wkcn can you please extend the date?

wkcn commented 1 year ago

HI @Andrewymd , thanks for your attention to our work! Here is the new link. The storage size of the teacher logits (clip_22k) is 499 GB.

Please install azcopy (https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) before downloading the teacher logits.

azcopy copy "https://hopengscus7387599778.blob.core.windows.net/tinyvit-teacher-logits/clip_224_22k_top100_logits_bs128/?sv=2021-08-06&se=2023-09-14T05%3A49%3A17Z&sr=c&sp=rl&sig=8HyBKYcLikVGtunLnTXgmtn8Ej0JiQsZt%2BCbexC6kfQ%3D" "./" --recursive
jotoy commented 11 months ago

Hi @wkcn can you please extend the date? Thanks for your great work~

wkcn commented 11 months ago

Hi @jotoy , here is the new download command.

azcopy copy  "https://hopengscus7387599778.blob.core.windows.net/tinyvit-teacher-logits/clip_224_22k_top100_logits_bs128/?sv=2021-08-06&st=2023-10-30T16%3A30%3A11Z&se=2023-11-29T16%3A30%3A00Z&sr=c&sp=rl&sig=MuAQWkYvVfD8q4YEKH2BdjdFoHR5ZkpKlMIOJlt2diM%3D" "./" --recursive

Thanks for your attention to our work!

maybeliuchuan commented 7 months ago

Hi @wkcn can you please extend the date? Thank you very much.

maybeliuchuan commented 7 months ago

Hi @wkcn, another question, since all of the 90epochs logits are too huge, whether can I train the TinyViT using only the logics of epoch 90? Thanks!

wkcn commented 7 months ago

@maybeliuchuan Thanks for your attention to our work!

Sorry that I could not extend the date. The logits can be reproduced by the teacher model CLIP-ViT-Large/14-22k on ImageNet-21k.

An epoch may be not enough on ImageNet-21k. You can try to save the teacher logits on ImageNet-1k. Here is the related tutorial: https://github.com/microsoft/Cream/blob/main/TinyViT/docs/SAVE_TEACHER_LOGITS.md