Fine-tuning details on iNaturalist dataset

facebookresearch / deit

Official DeiT repository

Apache License 2.0

4.03k stars 554 forks source link

Fine-tuning details on iNaturalist dataset #105

Closed Reinhard-Tichy closed 3 years ago

Reinhard-Tichy commented 3 years ago

In #45 , I have found some basic details about finetuning for CIFAR10 and Cars, but I could not find anything about other datasets.

So, could you please also provide the details about hyperparameters used in iNat2018 & 2019 for finetuning?

Thank you very much!

TouvronHugo commented 3 years ago

Hi @Reinhard-Tichy , Thanks for your question, The hparams for iNaturalist with the base model are:

Batch size: 768
LR: 7.5e-5
epochs: 360
Weight decay 0.05
Scheduler: cosine
Repeated augmentation: True
Smoothing: 0.1
Warmup: 5 epochs
RandAugment: rand-m9-mstd0.5-inc1
Mixup: 0.8
CutMix: 1.0
Stochastic Depth:0.1
Erasing: 0.1
Optimizer: AdamW Don't hesitate if you have any other questions, Best, Hugo

Reinhard-Tichy commented 3 years ago

Thanks for your reply! Could you further explain what the param Stochastic Depth means here? Cause I seemly cannot find this arg in the main.py

And does the finetuning process really need a 360 epochs? This is even more than the one of the pretraining step [Lol]

TouvronHugo commented 3 years ago

Hi @Reinhard-Tichy , For Stochastic Depth this is the --drop-path argument. Yes, this allows for better results but it is possible to fine-tune with much less epochs but the results will be slightly worse. Best, Hugo

TouvronHugo commented 2 years ago

FYI, I added the SGD config for iNat finetuning here