Closed JustinLiang closed 7 years ago
It appears that there was some sort of dependency issue with the server I was running it on.
Hi JustinLiang, I am also running the ResNet 101. It run well in traning without error. However, when I start to run the testing, it has error about memory
I0101 00:03:58.796654 15999 caffe.cpp:252] Running for 1449 iterations.
F0101 00:03:59.323478 15999 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7fdd0dd67daa (unknown)
@ 0x7fdd0dd67ce4 (unknown)
@ 0x7fdd0dd676e6 (unknown)
@ 0x7fdd0dd6a687 (unknown)
@ 0x7fdd0e3cadb8 caffe::SyncedMemory::to_gpu()
@ 0x7fdd0e3c9e89 caffe::SyncedMemory::mutable_gpu_data()
@ 0x7fdd0e392f62 caffe::Blob<>::mutable_gpu_data()
@ 0x7fdd0e47c008 caffe::BaseConvolutionLayer<>::forward_gpu_gemm()
@ 0x7fdd0e59c5c6 caffe::ConvolutionLayer<>::Forward_gpu()
@ 0x7fdd0e3a4dc2 caffe::Net<>::ForwardFromTo()
@ 0x7fdd0e3a4ed7 caffe::Net<>::ForwardPrefilled()
@ 0x4074f7 test()
@ 0x405c88 main
@ 0x7fdd0d378f45 (unknown)
@ 0x406327 (unknown)
@ (nil) (unknown)
Aborted (core dumped)
How many memory do we need to run the testing model? I do not know why training can run but testing cannot run. I am using TitanX pascal 12GB
+1 any solution?
mjohn123, are your training and testing images the same size? What size are they?
@JustinLiang, i am using original deeplab code. You are right. The training is 321 but testing is 531. Can i reduce it? Because the original image is more than 500
@JustinLiang I've run into the same 'out of memory' problem, and I've tried forwarding on a single image with very small size, the problem was still thrown, Is there any solution?
@lee2430 I got it to work in a machine with four TitanX (12Gb each one) but it's too expensive, I recommended you to try it in Tensorflow because it manages the memory better...
@amlarraz Thanks for your advice. I just see your comment posted 20 days ago today, but i've been learning tensorflow like what you said, cuz i'm a newbee to it :). Glad to know i'm in the right path.
@mjohn123 @amlarraz @JustinLiang @lee2430 Sorry for interrupting guys. I have run into the same "out of memory" issue. Just wondering if any of you were able to run the tests with ResNet-101 on machines with one TITAN X GPU? In addition, is it OK for me to simply reduce the test image dimension from 531 (in the test prototxt) to 321 (same as in the training prototxt)? If I reduce the image dimension is there anything else I should also change? Thanks!
@realwecan Hi, I tried a tensorflow version and it now works smoothly. https://github.com/DrSleep/tensorflow-deeplab-resnet
Sorry to bother you @JustinLiang . I am wondering why the training log outputs 12 output values while the network only produces 4 outputs? Why there are two 0s before the true accuracy?
I set up the DeepLabV2-ResNet101 model from the bitbucket code (so not using this python implementation) and during training I notice that the loss is going up. As I run this program for longer, the loss approaches 350. I was wondering if anyone would have any idea what could be the cause of this. Here is my run_pascal.sh script, I did not modify anything past the
## Training #1 (on train_aug)
comment:Furthermore, I have the
SegmentationClassAug
ground truth images from your dropbox link (https://www.dropbox.com/s/oeu149j8qtbs1x0/SegmentationClassAug.zip?dl=0) in thedata/VOCdevkit/VOC2012/
folder.Here is what the training looks like: