Closed blossomzx closed 5 years ago
Hi, it looks like a memory problem to me. If possible, resampling your image, by a factor between 0.5 and 1, should work in your case. Mini batch size of 64 is definitely too big, I'd normally use 2 - 8 pairs of images for a mini batch. Of course, it depends the size of the gpu you have.
On Mon, Oct 8, 2018, 15:36 Yang notifications@github.com wrote:
I used my image data(the sizes of the moving images and the fixed images are all 117x100x255 ) to train the network, and occurred the following problem:
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,117,100,255,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: local_up_0/additive_upsampling/resize_volume/transpose_1 = TransposeT=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
This problem still existed even I changed the minibatch_size to 1. Do I have to change the size of the images?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/YipengHu/label-reg/issues/9, or mute the thread https://github.com/notifications/unsubscribe-auth/ACd9_eyYZU_B7T1ZTydJ50k29HsZkhnNks5ui2LwgaJpZM4XNErX .
Thank you very much.
I used my image data(the sizes of the moving images and the fixed images are all 117x100x255 ) to train the network, and occurred the following problem:
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,117,100,255,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: local_up_0/additive_upsampling/resize_volume/transpose_1 = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](local_up_0/additive_upsampling/resize_volume/Reshape_3, local_up_0/additive_upsampling/resize_volume/transpose/perm)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
This problem still existed even I changed the minibatch_size to 1. Do I have to change the size of the images?