search

jcjohnson / densecap

Dense image captioning in Torch
MIT License
1.58k stars 430 forks source link

How much memory do I need to get the model run? #11

Closed liueryun closed 8 years ago

liueryun commented 8 years ago

My mac os x system has 16G RAM and 4G GPU memory. It seems still not enough to run the model.

$ th run_model.lua

THCudaCheck FAIL file=/Users/eryunliu/torch/extra/cutorch/lib/THC/generic/THCStorage.cu line=41 error=2 : out of memory /Users/eryunliu/torch/install/bin/luajit: /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:11: cuda runtime error (2) : out of memory at /Users/eryunliu/torch/extra/cutorch/lib/THC/generic/THCStorage.cu:41 stack traceback: [C]: in function 'resize' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:11: in function 'torch_Storage_type' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:57: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type' /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType' /Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type' ./densecap/DenseCapModel.lua:199: in function 'convert' run_model.lua:148: in main chunk [C]: in function 'dofile' ...nliu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x010b13bd50

jcjohnson commented 8 years ago

You are running out of GPU memory, not system memory. To run in GPU mode you need about 6GB of GPU memory.

liueryun commented 8 years ago

Thanks! @jcjohnson

jcjohnson commented 8 years ago

You can try running in CPU only mode by adding the flag -gpu -1

SeekPoint commented 8 years ago

**I got gtx1080 8G gpu card, but I ran with another program maybe take 1800M gpu memory and got this error.

I wiil try later to put all 8g mem to this program.**

Processed image 2395955.jpg (999 / 1000) of split 1, detected 86 regions
Processed image 2395949.jpg (1000 / 1000) of split 1, detected 102 regions
Loss stats: { mid_box_reg_loss : 0.0010207249870291 captioning_loss : 22.209516950607 end_box_reg_loss : 0.0019101264009718 end_objectness_loss : 0.040239814032614 total_loss : 22.343282449476 mid_objectness_loss : 0.090594833448529 } Average loss: 22.343282449476 IMG 30 PRED: a white and white line, GT: canopy has blue and white stripes. white and black awning falling down. , OK: 1, OV: 0.456080 SCORE: 0.141009
IMG 40 PRED: a man wearing a black shirt, GT: a white and green street bike. , OK: 1, OV: 0.241768 SCORE: 0.028571
IMG 50 PRED: a wooden fence, GT: the steps are stone. , OK: 1, OV: 0.237883 SCORE: 0.000000 IMG 80 PRED: the man is wearing a black pants, GT: surfers long sleeve black shirt. , OK: 1, OV: 0.387980 SCORE: 0.121905
IMG 110 PRED: a large brown tree, GT: ostriches have long necks. , OK: 1, OV: 0.248543 SCORE: 0.000000
IMG 121 PRED: a large tree in the background, GT: dark green trees on hill. , OK: 1, OV: 0.490476 SCORE: 0.000000
IMG 131 PRED: a black and black laptop, GT: black lamp on the right side of the room. , OK: 1, OV: 0.594925 SCORE: 0.067416 IMG 258 PRED: a man wearing a white shirt, GT: a baseball player. baseball player swinging a bat. baseball player wearing blue and gray. batter swinging the bat. a baseball player squatting with his arms and bat in the air. , OK: 1, OV: 0.498223 SCORE: 0.076433
IMG 345 PRED: a train on the train, GT: train tracks. , OK: 1, OV: 0.117586 SCORE: 0.186047 IMG 376 PRED: the sky is blue, GT: the sky is cloudy. , OK: 1, OV: 0.427383 SCORE: 0.323972 IMG 439 PRED: a man wearing a white shirt, GT: child sitting in bottom row far left. , OK: 1, OV: 0.415828 SCORE: 0.000000
IMG 492 PRED: the front of the bus is red, GT: black writing on the white license plate. , OK: 1, OV: 0.308367 SCORE: 0.024390
IMG 523 PRED: a wooden table, GT: a tall thin microphone. a black microphone stand. , OK: 1, OV: 0.144778 SCORE: 0.041885
IMG 638 PRED: the shirt is white, GT: the shirt is plaid. , OK: 1, OV: 0.347591 SCORE: 0.323972 IMG 648 PRED: the ground is made of the water, GT: wooden plank deck. , OK: 1, OV: 0.306353 SCORE: 0.000000 IMG 767 PRED: a white sink, GT: this is a metal. , OK: 1, OV: 0.376215 SCORE: 0.065041
IMG 788 PRED: a white and white plate, GT: a wet nap from deli. , OK: 1, OV: 0.605932 SCORE: 0.021448 IMG 819 PRED: a white wall, GT: knives inside a glass case. rubber ducks atop a knife case. a box of wood. a display box with a glass front. , OK: 1, OV: 0.801896 SCORE: 0.033058
IMG 861 PRED: the sky is blue, GT: it is foggy in the city. , OK: 1, OV: 0.367248 SCORE: 0.082474
IMG 872 PRED: a large street, GT: a platform outside of a building. , OK: 1, OV: 0.317051 SCORE: 0.035556
IMG 923 PRED: the sign is red, GT: a stop sign is visible. , OK: 1, OV: 0.427752 SCORE: 0.181107
IMG 944 PRED: a light on the wall, GT: a ladder on the bunk bed. , OK: 1, OV: 0.448346 SCORE: 0.116035
IMG 955 PRED: a black and white motorcycle, GT: this is a sidecar. , OK: 1, OV: 0.552734 SCORE: 0.059259
mAP: 3.937982
wrote checkpoint.t7.json
Warning: cudnn.convert does not work with nngraph yet. Ignoring nn.gModulewrote checkpoint.t7
Warning: cudnn.convert does not work with nngraph yet. Ignoring nn.gModuleTHCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-6130/cutorch/lib/THC/generic/THCStorage.cu line=65 error=2 : out of memory /home/rzai/torch/install/bin/luajit: /home/rzai/torch/install/share/lua/5.1/nn/Module.lua:301: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-6130/cutorch/lib/THC/generic/THCStorage.cu:65 stack traceback: [C]: in function 'Tensor' /home/rzai/torch/install/share/lua/5.1/nn/Module.lua:301: in function 'flatten' /home/rzai/torch/install/share/lua/5.1/nn/Module.lua:318: in function 'getParameters' ./densecap/DenseCapModel.lua:374: in function 'getParameters' train.lua:196: in main chunk [C]: in function 'dofile' ...rzai/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670 rzai@rzai00:/media/rzai/ai_data/prj/densecap$ rzai@rzai00:/media/rzai/ai_data/prj/densecap$