search

ASMIftekhar / VSGNet

VSGNet:Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions.
MIT License
100 stars 20 forks source link

Why is the GPU util rate very low when training the VSGNet #7

Closed truetone2022 closed 4 years ago

truetone2022 commented 4 years ago
image
ASMIftekhar commented 4 years ago

Thanks a lot for the question. Without knowing other information, like batch size, GPU type I don't think I can answer this question.

Just to let you know, for V-COCO with a batch size of 8 in a RTX 2080TI each epoch takes close to 10 mints and as mentioned in the repo for hico it takes close to 40 mints in 4 RTX 2080TIs with a batch size of 64. One thing to point out, we work with a flexible number of humans and objects, so the usage of GPU changes heavily during different iterations.

truetone2022 commented 4 years ago

Thanks a lot for your kind reply! It's very helpful! I also set a batch size of 64 in 4 RTX 2080TIs for HICO-DET, the GPU memory is almost run out, but the gpu util rate is less than 30% all the time.Were you facing the same situation when training the VSGNet? Very thanks for any helpful reply~Merci beaucoup~

ASMIftekhar commented 4 years ago

I cant recall the gpu use actually. But if it takes close to 40 mints then I would say there is nothing to get worried.

truetone2022 commented 4 years ago

Very thanks for your kind reply!

truetone2022 commented 4 years ago

Should the learning rate be set to 0.001*8 when the batch size is 64 in the procedure of training VSGNet on HICO-DET ? Very thanks for any helpful reply!

ASMIftekhar commented 4 years ago

Without running the model I can't say anything in particular but as per as I remember for HICO our model was converging fine with different learning rates.