Not working for 1080x1080 images

Engineering-Course / CIHP_PGN

Code repository for Part Grouping Network, ECCV 2018

MIT License

430 stars 117 forks source link

Not working for 1080x1080 images #31

Open rishilss99 opened 5 years ago

rishilss99 commented 5 years ago

The pre-trained model works for smaller images(300,200) but when training with (1080,1080) images gives the error: OP_REQUIRES failed at concat_op.cc:153 : Resource exhausted: OOM when allocating tensor with shape[2,2560,169,169] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Please suggest a solution as I need to use 1080x1080 image for 3D reconstruction. I am working with 4 GPUs and enough RAM for the process.

DomhnallBoyle commented 5 years ago

Having the same issue, @Engineering-Course is there a possible solution for this?

YifuTao commented 4 years ago

So the code fails for multiple 1080x1080 images withe the aforementioned error but works with only one image. Modify val_id.txt and val.txt with only one image. Think it is possible to modify the code to store images one by one instead of all together and crash the GPU memory.

lll-gen commented 4 years ago

I have the same question，how to solve this problem?