search

elchun / contact_graspnet_pytorch

Pytorch implementation of Contact-GraspNet
Other
18 stars 5 forks source link

About the training time #3

Open FredTrumpSenior opened 6 months ago

FredTrumpSenior commented 6 months ago

Thanks for your endeavor of this work. However, I came into a problem when I trained the model. How long will it take to train the model for the default 200 epochs? I used a single 2080Ti GPU for about 1100 hours, about 40 days, and 8 cards in parallel for about 200+ hours, far more than the official tensorflow version claims to train 44 hours on a V100. Is this normal?

elchun commented 5 months ago

Hello, we found that the performance leveled off in about a day or two. Perhaps you could check the validation loss or run the model on a sample scene and stop training when this doesn't change much. By default, I believe that the training loop does not automatically stop.