yitu-opensource / T2T-ViT

ICCV2021, Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Other
1.14k stars 177 forks source link

small question about lr_scheduler #38

Closed huixiancheng closed 3 years ago

huixiancheng commented 3 years ago

Thanks for the opensource code!! Could you tell me the meaning of metric in lr step? https://github.com/yitu-opensource/T2T-ViT/blob/f436fe4043069989ec5e0c2d07407b6d898493a7/main.py#L577-L579 https://github.com/yitu-opensource/T2T-ViT/blob/f436fe4043069989ec5e0c2d07407b6d898493a7/main.py#L688-L689

In my understanding.Look like in timm it's don't have special meaning.

yuanli2333 commented 3 years ago

Hi, lr_scheduler is the method to set up lr decay in training, such as multi-step decay and cosine decay. We use cosine decay in this work.

huixiancheng commented 3 years ago

I know it's cosine_schedule with warm up. But the ques is about update the ls after a epoch or batch(iter), there are two paras: epoch/num_updates and metric. Since I don't use timm before,I cann't understand the meaning of Para:metric In timm it don't have special meaning too. As the code show. QQ截图20210325112532

yuanli2333 commented 3 years ago

You need update ls in each epoch to tell it current epoch, which is the ls usage in torch.