search

apple / ml-cvnets

CVNets: A library for training computer vision networks
https://apple.github.io/ml-cvnets
Other
1.77k stars 225 forks source link

Training Time on ImageNet #50

Closed StephenEkaputra closed 1 year ago

StephenEkaputra commented 2 years ago

Hi, I wonder how long does it take for you to train MobileViT-S with 8 GPUs? I trained your model MobileViT-S with 1024 batch size (128*8) for 1 epoch with 8 V100 GPUs, but the training time is very slow. It costs like 40 minutes/epoch. For 300 epochs, it means more than 8 days. Is it normal?

Thank you

iantimmis commented 2 years ago

Try with 8 A100s. It will be a lot faster, and per-model-trained will be cheaper.

sacmehta commented 1 year ago

Hi, We did not train on V100 GPUs, so no idea about training time on these GPUs.

We trained on A100 GPUs and it takes about 1.5-2 days for training.

StephenEkaputra commented 1 year ago

Thank you for your reply