Closed lzhnb closed 4 years ago
Suppose dataset is fine. When i train, i got the memory error like this
Loaded dataset `voc_2012_train_first_split` for training
Set proposal method: gt
Appending horizontally-flipped training examples...
wrote gt roidb to /mnt/Disk1/zhliang/code/MetaR-CNN/data/cache/voc_2012_train_first_split_gt_roidb.pkl
done
Preparing training data...
done
before filtering, there are 12330 images...
after filtering, there are 12330 images...
12330 roidb entries
Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth
THCudaCheck FAIL file=/pytorch/torch/lib/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "train_metarcnn.py", line 384, in <module>
num_boxes_list)
File "/home/lzhnb/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
result = self.forward(*input, **kwargs)
File "/mnt/Disk1/zhliang/code/MetaR-CNN/lib/model/faster_rcnn/faster_rcnn.py", line 61, in forward
attentions = self.prn_network(prn_data)
File "/mnt/Disk1/zhliang/code/MetaR-CNN/lib/model/faster_rcnn/resnet.py", line 328, in prn_network
base_feat = self.RCNN_base(self.meta_conv1(im_data))
...
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58
And this error happened in fp(lib/model/fasterrcnn/resnet.py)
def prn_network(self,im_data):
'''
the Predictor-head Remodeling Network (PRN)
:param im_data:
:return attention vectors:
'''
base_feat = self.RCNN_base(self.meta_conv1(im_data))
feature = self._head_to_tail(self.max_pooled(base_feat))
attentions = self.sigmoid(feature)
return attentions
My server is 8700k+titan xp(12 G)2 and 16g2 with ubuntu 16.04 cuda8.0+cudn7.1.4 and the torch==0.3.1, torchvision==0.2.0
Is there any question in running train?
Thanks
In the first phase, the metadata get the shots from the imgset voc2007 and voc2012, so the shots are more than 200. In the second, we set the self.shots = 3 * shots from the imgset voc2007. I have updated the code. As for problem of out of memory, you can reduce the batch_size.
In the first phase, the metadata get the shots from the imgset voc2007 and voc2012, so the shots are more than 200. In the second, we set the self.shots = 3 * shots from the imgset voc2007. I have updated the code. As for problem of out of memory, you can reduce the batch_size.
Could you explain the purpose of self.shots= 3*shots you set?
I make everything ok and tun the train
train_metarcnn.py
But i got error like this:
located in here (datasets/metadata.py around line58):
Because the train_metarcnn give the
args.phase=1
it will give theMetaDataset
initialization parameter-shots=200
And then I print the parameters:
I find that
For different class, the number of its image is different and some of them less than 200, so it must be out of range.
In your papre the $D{meat}$ and $D{train}$ are exclude, so I guess you want to load some image into dataset and i change the code, if we need 200shots and there are 15 classes in $C_{meat}$ for VOC dataset, so we collect around 13 shots from each class. So i change the code and it runs.
Isn't it right? And i found the
self.shots
is three times larger thanshots
. Maybe i miss some detail of the paper, I will read it again.Thanks