Finetuning fails on ViG

[tdzdog]

I tried to finetune on ViG but the accuracy keeps decreasing. I used the ViG tiny and the provided vig_ti_74.5.pth as the pretrained weights. I finetuned for 30 epochs and used first 5 epochs for warmup. The accuracy decreases even in the warmup stage (74.5%->73.9%). Then the accuracy even decreases to 62.5%. I used similar learning rate 2e-3 and cosine scheduler (same as pretraining). I am wondering what is going wrong?

[iamhankai]

During finetuing, the learning rate is usually set smaller (e.g., x0.1) than during pretraining.