Technical Problem of Testing the Stanford40 Dataset

[whq-hqw]

Hi, everyone. I tested the pascal-voc 2012 datasets, RstarCNN works great! But there's a little problem when testing on Stanford40.

• In the ./tools/test_net.py : (line 13)from fast_rcnn.test import test_net --> from fast_rcnn.test_stanford40 import test_net (line 16)import datasets.stanford40 was added. (line 78)imdb = datasets.pascal_voc('val','2012')--> imdb = datasets.stanford40('test')
• In the ./lib/datasets/stanford40.py : (line 295)datasets.pascal_voc('trainval', '2012')--> d = datasets.stanford40('test')

After that, I typed ./tools/test_net.py --gpu 0 --def models/VGG16_RstarCNN_stanford40/test.prototxt --net ./trained_models/stanford40_rstarcnn_train.caffemodel into the terminal, and finally: Network initialization done and Memory required for data: 117424480 appeared. (This is satisfying output)

Then comes the problem, the terminal says: File "./tools/test_net.py", line 78, in <module> imdb = datasets.stanford40('test') TypeError: 'module' object is not callable

I tried to figure out the problem myself, so I add the following code to ./tools/test_net.py: print(type(datasets.pascal_voc)), the print result turn out to be: <type 'type'> print(type(datasets.stanford40)), the print result turn out to be: <type 'module'>

In ./lib/datasets/"pascal_voc.py" & "stanford40.py" are written in almost the same manner, I don't know what cause the problem. If I missed something please correct me.

[whq-hqw]

It seems the previous problem can be fixed by simply copy the code form stanford40.py into pascal_voc.py. (Anyone knows why?)

Now There's a memory problem during Testing. Every time when testing the 1024/5532 image, the terminal says: im_detect: 1020/5532 0.382s im_detect: 1021/5532 0.382s im_detect: 1022/5532 0.382s im_detect: 1023/5532 0.382s im_detect: 1024/5532 0.383s F0117 14:23:43.965023 9983 syncedmem.cpp:58] Check failed: error == cudaSuccess (2 vs. 0) out of memory *** Check failure stack trace: *** Aborted (core dumped)

I tried to monitor the memory usage of my 8GB GTX 1080, the memory usage just keep increasing during testing phase, maybe there's some modification that I should but haven't done, any advice?

[gkioxari]

A wild guess from this snapshot is that the 1025th image is large and due to the resize that happens when processing the input, the network runs out of memory.

If you resolved this issue, could you close the issue?

[whq-hqw]

Unfortunately no, the GPU memory usage is keep climbing during the test, I use nvidia-smi to monitor the memory consumption and the result is below: TEST Iter. PID TYPE PROCESS MEM USAGE 100 30124 C python 6319MiB 200 30124 C python 6497MiB 300 30124 C python 6653MiB 400 30124 C python 6653MiB 500 30124 C python 6653MiB 600 30124 C python 6681MiB 700 30124 C python 6681MiB 800 30124 C python 6821MiB 900 30124 C python 6821MiB 1000 30124 C python 7277MiB

I'd like to know the where I can do some memory management in RstarCNN Project. Any advice would be appreciated, thank you.

[gkioxari]

You should play with the scale of the input images at training and testing time. Those are set in the config file. The convolutions on the image during the initial layers are the most expensive operations. So decreasing the size of the images should solve your memory issue. However, this will come at a performance cost. You will be effectively decreasing the resolution of your input. If you end up decreasing the image resolution, I'd advise you to retrain your model with that resolution for better performance.

[whq-hqw]

Thank you very much

[maggiesh]

About the error below, I solved it by adding "from .stanford40 import stanford40" into lib/datasets/init.py . JFYI .

File "./tools/test_net.py", line 78, in imdb = datasets.stanford40('test') TypeError: 'module' object is not callable