byaman14
commented
2 years ago Hi Zhengguo,
This issue occurs due to GPU memory limitations. The data you are trying to fit is very large. There are several things you can do to resolve this issue.
1) Remove oversampling in your data. The dataset you have oversampled along kx direction. You can remove oversampling which is a common practice in DL based MRI reconstruction. In your case, after removing oversampling, kspace/ sensitivity maps will have size of nSlices x 320x320 xncoils and mask should have size of 320x320. I expect this to resolve the issue.
2) You can reduce the complexity of the network by reducing the number of unrolling networks as well as number of unroll blocks.
ZhengguoTan
Dear Dr. Yaman,
I managed to setup the kspace_dir, coil_dir, and mask_dir, and start to run train.py.
However, it seems some problem occurs at https://github.com/byaman14/SSDU/blob/main/train.py#L136
Here are the terminal output I got: " ... Normalize the kspace to 0-1 region
size of kspace: (16, 640, 320, 20) , maps: (16, 640, 320, 20) , mask: (640, 320)
create training and loss masks and generate network inputs...
Iteration: 0
Gaussian selection is processing, rho = 0.40, center of kspace: center-kx: 319, center-ky: 160
size of ref kspace: (16, 20, 640, 320, 2) , nw_input: (16, 640, 320, 2) , maps: (16, 20, 640, 320) , mask: (16, 640, 320) SSDU Parameters: Epochs: 100 , Batch Size: 1 , Number of trainable parameters: 592129 Training... Traceback (most recent call last): File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[20,640,160] and type complex64 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[{{node SSDUModel/Weights/mapCG_1/while/CGloop/CGIters/EhE/strided_slice_3}}]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "train.py", line 140, in
tmp, , = sess.run([loss, update_ops, optimizer])
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[20,640,160] and type complex64 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node SSDUModel/Weights/mapCG_1/while/CGloop/CGIters/EhE/strided_slice_3 (defined at /home/hpc/iwbi/iwbi005h/Softwares/SSDU/tf_utils.py:125) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'SSDUModel/Weights/mapCG_1/while/CGloop/CGIters/EhE/strided_slice_3', defined at: File "train.py", line 113, in
nw_output_img, nw_outputkspace, * = UnrollNet.UnrolledNet(nw_input_tensor, sens_maps_tensor, trn_mask_tensor, loss_mask_tensor).model
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/UnrollNet.py", line 43, in init
self.model = self.Unrolled_SSDU()
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/UnrollNet.py", line 61, in Unrolled_SSDU
x = ssdu_dc.dc_block(rhs, self.sens_maps, self.trn_mask, mu)
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/data_consistency.py", line 121, in dc_block
dc_block_output = tf.map_fn(cg_map_func, (rhs, sens_maps, mask), dtype=tf.float32, name='mapCG')
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/functional_ops.py", line 497, in map_fn
maximum_iterations=n)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop
return_same_structure)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop
body_result = body(packed_vars_for_body)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3525, in
body = lambda i, lv: (i + 1, orig_body( lv))
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/functional_ops.py", line 486, in compute
packed_fn_values = fn(packed_values)
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/data_consistency.py", line 117, in cg_map_func
cg_output = conj_grad(input_elems, mu)
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/data_consistency.py", line 106, in conj_grad
cg_out = tf.while_loop(cond, body, loop_vars, name='CGloop', parallel_iterations=1)[2]
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3556, in while_loop
return_same_structure)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3087, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3022, in _BuildLoop
body_result = body(packed_vars_for_body)
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/data_consistency.py", line 91, in body
Ap = Encoder.EhE_Op(p, mu)
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/data_consistency.py", line 30, in EhE_Op
kspace = tf_utils.tf_fftshift(tf.fft2d(tf_utils.tf_ifftshift(coil_imgs))) / self.scalar
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/tf_utils.py", line 154, in tf_ifftshift
return tf_ifftshift_flip2D(tf_ifftshift_flip2D(input_x, axes=1), axes=2)
File "/home/hpc/iwbi/iwbi005h/Softwares/SSDU/tf_utils.py", line 125, in tf_ifftshift_flip2D
second_half = tf.identity(input_data[:, :, ny:])
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 654, in _slice_helper
name=name)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 820, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 9356, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(args, **kwargs)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/home/hpc/iwbi/iwbi005h/.conda/envs/ssdu/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[20,640,160] and type complex64 on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node SSDUModel/Weights/mapCG_1/while/CGloop/CGIters/EhE/strided_slice_3 (defined at /home/hpc/iwbi/iwbi005h/Softwares/SSDU/tf_utils.py:125) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. ... "
It reads quite complicated to myself. Would you know what the problem might be here?
Thank you in advance for your help!
Best Regards, Zhengguo