search

yuleiniu / vc

Code for CVPR'18 "Grounding Referring Expressions in Images by Variational Context"
29 stars 4 forks source link

Hi，When I train the network, the used GPU memery keep going up? #1

Open leoozy opened 6 years ago

leoozy commented 6 years ago

Hi, Thank for your nice code. It is really a beautiful project. When I am trying to train the dataset refcoco, the used GPU memery keep going up. Initially, the used memery is about 4.8GB. After about 1000 iters, there used memery has been added up to 9.2GB. And I read the code carefully, you feed the net one image in every batch (I am not sure). But why the batchsize is constant, but the GPU memery used keep going up? My GPU is : GTX1080 TI, 11GB. I want to know is this your final code? And have you ever meet this error? Thank you!

yuleiniu commented 6 years ago

Hi, the problem is that the number of bounding boxes and referring expressions (sentences) varies in different images. You can try to feed only one sentence per iteration, which can help to reduce GPU memory but slightly hurt performance. It should be noted that the batch size in our code is defined over sentences rather than images.

leoozy commented 6 years ago

Thank you, I fixed it with two 1080Ti

leoozy commented 6 years ago

Hi, I may get some problems when training the unsupervised setting. Everything is fine in the supervised setting. While in unsupervised setting, the result goes wrong: image I train the model follow your command on the refcoco datasets. The printed [Nan ...] is the value of scores_val. Could you please tell me how to fix it?

yuleiniu commented 6 years ago

I think the problem might be computing log(0) in unsupervised setting. I have modified vc_model.py to prevent this.