search

rbgirshick / py-faster-rcnn

Faster R-CNN (Python implementation) -- see https://github.com/ShaoqingRen/faster_rcnn for the official MATLAB version
Other
8.1k stars 4.11k forks source link

Too much virtual memory use in end2end training #147

Open ZhijingX opened 8 years ago

ZhijingX commented 8 years ago

Hi, I am running the end2end training code for only Pascal 2007 dataset using ZF model. But when it starts, the machine starts to function really slow. One core of CPU (8 core) is running. GPU (Titan X, 12G memroy) is running full-speed. Memory (50% used) and swap (0% used) functioning normally. Only problem is when I type "top" in the terminal, the VIRT (virtual memory) was taken up about 70g by python. The training speed is about 4s/iteration. Is this normal? Anyone has the same issue?

duygusar commented 8 years ago

Yes, as soon as I run training, python goes crazy (I have a Tesla C2070 5376 MB dedicated memory and I am training ZF, batchsizes set to 1 ):

27361 root 20 0 58.416g(VIRT) 123480 53816 R 148.6 (%CPU - then it settles around 99 and goes out of memory) 0.3 0:04.48 python

and this might be causing my problem with the memory https://github.com/rbgirshick/py-faster-rcnn/issues/253

duygusar commented 8 years ago

@Paralysis Have you ever figured out a way around to decrease the memory consumption? or do you have a clue what is causing it?

xiaoxiongli commented 8 years ago

I use about 20W+ images to train, and i find that it use about 500G virtual memory!!!