search

taigw / brats17

Brain tumor segmentation for MICCAI 2017 BraTS challenge
BSD 3-Clause "New" or "Revised" License
324 stars 130 forks source link

Memory error, while training after data loading. #17

Closed charan223 closed 6 years ago

charan223 commented 6 years ago

Are multiple GPU's supported? Getting same error with single, multiple GPU's (8gb each)

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[5,17,72,72,32] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: gradients/MSNet_WT32/block2_2/prelu_acti_0/Abs_grad/Sign = SignT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[Node: gradients/loss_function/mul_6_grad/tuple/group_deps/_754 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4124_gradients/loss_function/mul_6_grad/tuple/group_deps", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]