Vision Transformer Learning Rate Adjustment

[cdreetz]

The 'Fine-tune the Vision Transformer on CIFAR-10 with PyTorch Lightning' tutorial notebook uses a learning rate of lr=5e-5 but should be changed to 2e-5 at the most but probably more like 1e-5. The provided lr won't get past ~5 epochs because of the early stop plus high learning rate.

After 5 runs with 5e-5 never got better than 94% val acc Running 2e-5 and then 1e-5 both ranged between 96-97% val acc

https://github.com/NielsRogge/Transformers-Tutorials/blob/master/VisionTransformer/Fine_tuning_the_Vision_Transformer_on_CIFAR_10_with_PyTorch_Lightning.ipynb

[cdreetz]

@NielsRogge Do you want a PR for this

[cdreetz]