I am using vit tiny as well as deit tiny with around 55k images on 2 gpus, with below parameters, the training is very slow, I could complete around 8 epochs in 24 hours. How can I make the training faster? I would also want to add more images as I have around 10 times bigger final dataset. I am starting with 55k images first but training with this only is very slow. Can someone suggest what parameters can help the training faster? I tried 4 GPU's but I think on our cluster 4 GPU configuration did not work out and the job got hung up after a couple of hours,so 2 GPU's is the max I can use.
hello,
I am using vit tiny as well as deit tiny with around 55k images on 2 gpus, with below parameters, the training is very slow, I could complete around 8 epochs in 24 hours. How can I make the training faster? I would also want to add more images as I have around 10 times bigger final dataset. I am starting with 55k images first but training with this only is very slow. Can someone suggest what parameters can help the training faster? I tried 4 GPU's but I think on our cluster 4 GPU configuration did not work out and the job got hung up after a couple of hours,so 2 GPU's is the max I can use.
Thanks, Rohan