I am trying to train model with brats18 data. If you could please help debugging an error.
The error generates while executing train.py file at line "dice = loss.eval(feed_dict ={x:tempx, w:tempw, y:tempy})"
During "step"=0 it works fine, and then at "step"=1, it gives error.
Below is the traceback:
(myEnv) PS C:\Users\Jayendra\Desktop\brats17-master\brats17-master> python .\train.py .\config17\train_wt_ax.txt
C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:469: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:470: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:471: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:472: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:473: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:476: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
data data_root C:\Users\Jayendra\Desktop\brats17-master\brats17-master\data_root\Brats17TrainingData C:\Users\Jayendra\Desktop\brats17-master\brats17-master\data_root\Brats17TrainingData
data data_names config17\train_names_all.txt config17\train_names_all.txt
data modality_postfix [flair, t1, t1ce, t2] ['flair', 't1', 't1ce', 't2']
data label_postfix seg seg
data file_postfix nii.gz nii.gz
data with_ground_truth True True
data batch_size 5 5
data data_shape [19, 144, 144, 4] [19, 144, 144, 4]
data label_shape [11, 144, 144, 1] [11, 144, 144, 1]
data label_convert_source [0, 1, 2, 3] [0, 1, 2, 3]
data label_convert_target [0, 1, 1, 1] [0, 1, 1, 1]
data batch_slice_direction axial axial
data train_with_roi_patch False False
data label_roi_mask None
data roi_patch_margin None
network net_type MSNet MSNet
network net_name MSNet_WT32 MSNet_WT32
network downsample_twice True True
network class_num 2 2
training learning_rate 1e-3 0.001
training decay 1e-7 1e-07
training maximal_iteration 20000 20000
training snapshot_iteration 5000 5000
training start_iteration 0 0
training test_iteration 100 100
training test_step 10 10
training model_pre_trained None
training model_save_prefix C:\Users\Jayendra\Desktop\brats17-master\brats17-master\model17\msnet_wt32 C:\Users\Jayendra\Desktop\brats17-master\brats17-master\model17\msnet_wt32
size of predicty: Tensor("MSNet_WT32/final_pred/conv:0", shape=(5, 11, 144, 144, 2), dtype=float32)
0.001
jay_before_opt_step
name: "Adam"
op: "NoOp"
input: "^Adam/update_MSNet_WT32/block1_1/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_1/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block1_2/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse1_bnprelu/conv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse1_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse1_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse1_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample1_bnprelu/conv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample1_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample1_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample1_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_1/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block2_2/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse2_bnprelu/conv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse2_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse2_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse2_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample2_bnprelu/conv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample2_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample2_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/downsample2_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_1/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_2/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block3_3/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse3_bnprelu/conv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse3_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse3_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse3_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_1/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_2/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/bn_0/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/bn_0/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/prelu_acti_0/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/conv_0/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/bn_1/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/bn_1/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/prelu_acti_1/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/block4_3/conv_1/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse4_bnprelu/conv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse4_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse4_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/fuse4_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up1_bnprelu/deconv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up1_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up1_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up1_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_1_bnprelu/deconv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_1_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_1_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_1_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_2_bnprelu/deconv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_2_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_2_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up2_2_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_1_bnprelu/deconv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_1_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_1_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_1_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_2_bnprelu/deconv/w/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_2_bnprelu/bn/beta/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_2_bnprelu/bn/gamma/ApplyAdam"
input: "^Adam/update_MSNet_WT32/pred_up3_2_bn_prelu/preluacti/alpha/ApplyAdam"
input: "^Adam/update_MSNet_WT32/final_pred/w/ApplyAdam"
input: "^Adam/Assign"
input: "^Adam/Assign_1"
jay_after_opt_step
2020-05-30 11:45:22.845647: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2020-05-30 11:45:23.401603: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:17:00.0
totalMemory: 16.00GiB freeMemory: 13.32GiB
2020-05-30 11:45:23.608883: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 1 with properties:
name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:73:00.0
totalMemory: 16.00GiB freeMemory: 13.32GiB
2020-05-30 11:45:23.801746: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 2 with properties:
name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:a6:00.0
totalMemory: 16.00GiB freeMemory: 13.32GiB
2020-05-30 11:45:23.821872: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Device peer to peer matrix
2020-05-30 11:45:23.831541: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1051] DMA: 0 1 2
2020-05-30 11:45:23.839970: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1061] 0: Y N N
2020-05-30 11:45:23.846304: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1061] 1: N Y N
2020-05-30 11:45:23.853917: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1061] 2: N N Y
2020-05-30 11:45:23.859826: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro RTX 5000, pci bus id: 0000:17:00.0, compute capability: 7.5)
2020-05-30 11:45:23.870241: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Quadro RTX 5000, pci bus id: 0000:73:00.0, compute capability: 7.5)
2020-05-30 11:45:23.877451: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: Quadro RTX 5000, pci bus id: 0000:a6:00.0, compute capability: 7.5)
jay_before_sess
<tensorflow.python.client.session.InteractiveSession object at 0x000001CA5DB267B8>
jay_after_sess
<tensorflow.python.training.saver.Saver object at 0x000001CA5DB5FE48>
Data load, 100.0% finished
20000
5000
0
100
10
0
0
100
10
0 ("this is value of "step")
jay_after_step
0.534315 ("this is dice value for "step" 0")
1 ("this is value of "step")
Traceback (most recent call last):
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
return fn(*args)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
status, run_metadata)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape
[[Node: loss_function/mul = SparseDenseCwiseMul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_function/stack, loss_function/ones_like, loss_function/ToInt64_2, loss_function/Softmax)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".\train.py", line 144, in
train(config_file)
File ".\train.py", line 123, in train
dice = loss.eval(feed_dict ={x:tempx, w:tempw, y:tempy})
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 570, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 4455, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
run_metadata_ptr)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
options, run_metadata)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape
[[Node: loss_function/mul = SparseDenseCwiseMul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_function/stack, loss_function/ones_like, loss_function/ToInt64_2, loss_function/Softmax)]]
Caused by op 'loss_function/mul', defined at:
File ".\train.py", line 144, in
train(config_file)
File ".\train.py", line 71, in train
loss = loss_func(predicty, y, weight_map = w)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\niftynet\layer\base_layer.py", line 32, in call
return self._op(*args, *kwargs)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\template.py", line 268, in call
result = self._call_func(args, kwargs, check_for_new_variables=False)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\template.py", line 217, in _call_func
result = self._func(args, *kwargs)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\niftynet\layer\loss_segmentation.py", line 61, in layer_op
pred, ground_truth, weight_map))
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\niftynet\layer\loss_segmentation.py", line 320, in dice
one_hot prediction, reduction_axes=[0])
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\math_ops.py", line 905, in binary_op_wrapper_sparse
name=name), sp_x.dense_shape)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\gen_sparse_ops.py", line 851, in sparse_dense_cwise_mul
sp_shape=sp_shape, dense=dense, name=name)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape
[[Node: loss_function/mul = SparseDenseCwiseMul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_function/stack, loss_function/ones_like, loss_function/ToInt64_2, loss_function/Softmax)]]
Hi,
I am trying to train model with brats18 data. If you could please help debugging an error.
The error generates while executing train.py file at line "dice = loss.eval(feed_dict ={x:tempx, w:tempw, y:tempy})" During "step"=0 it works fine, and then at "step"=1, it gives error.
Below is the traceback: (myEnv) PS C:\Users\Jayendra\Desktop\brats17-master\brats17-master> python .\train.py .\config17\train_wt_ax.txt C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:469: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:470: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:471: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:472: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:473: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\dtypes.py:476: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) data data_root C:\Users\Jayendra\Desktop\brats17-master\brats17-master\data_root\Brats17TrainingData C:\Users\Jayendra\Desktop\brats17-master\brats17-master\data_root\Brats17TrainingData data data_names config17\train_names_all.txt config17\train_names_all.txt data modality_postfix [flair, t1, t1ce, t2] ['flair', 't1', 't1ce', 't2'] data label_postfix seg seg data file_postfix nii.gz nii.gz data with_ground_truth True True data batch_size 5 5 data data_shape [19, 144, 144, 4] [19, 144, 144, 4] data label_shape [11, 144, 144, 1] [11, 144, 144, 1] data label_convert_source [0, 1, 2, 3] [0, 1, 2, 3] data label_convert_target [0, 1, 1, 1] [0, 1, 1, 1] data batch_slice_direction axial axial data train_with_roi_patch False False data label_roi_mask None data roi_patch_margin None network net_type MSNet MSNet network net_name MSNet_WT32 MSNet_WT32 network downsample_twice True True network class_num 2 2 training learning_rate 1e-3 0.001 training decay 1e-7 1e-07 training maximal_iteration 20000 20000 training snapshot_iteration 5000 5000 training start_iteration 0 0 training test_iteration 100 100 training test_step 10 10 training model_pre_trained None training model_save_prefix C:\Users\Jayendra\Desktop\brats17-master\brats17-master\model17\msnet_wt32 C:\Users\Jayendra\Desktop\brats17-master\brats17-master\model17\msnet_wt32 size of predicty: Tensor("MSNet_WT32/final_pred/conv:0", shape=(5, 11, 144, 144, 2), dtype=float32) 0.001 jay_before_opt_step name: "Adam" op: "NoOp" input: "^Adam/update_MSNet_WT32/block1_1/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_1/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block1_2/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse1_bnprelu/conv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse1_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse1_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse1_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample1_bnprelu/conv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample1_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample1_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample1_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_1/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block2_2/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse2_bnprelu/conv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse2_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse2_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse2_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample2_bnprelu/conv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample2_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample2_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/downsample2_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_1/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_2/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block3_3/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse3_bnprelu/conv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse3_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse3_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse3_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_1/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_2/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/bn_0/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/bn_0/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/prelu_acti_0/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/conv_0/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/bn_1/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/bn_1/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/prelu_acti_1/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/block4_3/conv_1/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse4_bnprelu/conv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse4_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse4_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/fuse4_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up1_bnprelu/deconv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up1_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up1_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up1_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_1_bnprelu/deconv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_1_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_1_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_1_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_2_bnprelu/deconv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_2_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_2_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up2_2_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_1_bnprelu/deconv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_1_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_1_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_1_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_2_bnprelu/deconv/w/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_2_bnprelu/bn/beta/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_2_bnprelu/bn/gamma/ApplyAdam" input: "^Adam/update_MSNet_WT32/pred_up3_2_bn_prelu/preluacti/alpha/ApplyAdam" input: "^Adam/update_MSNet_WT32/final_pred/w/ApplyAdam" input: "^Adam/Assign" input: "^Adam/Assign_1"
jay_after_opt_step 2020-05-30 11:45:22.845647: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2 2020-05-30 11:45:23.401603: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties: name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815 pciBusID: 0000:17:00.0 totalMemory: 16.00GiB freeMemory: 13.32GiB 2020-05-30 11:45:23.608883: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 1 with properties: name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815 pciBusID: 0000:73:00.0 totalMemory: 16.00GiB freeMemory: 13.32GiB 2020-05-30 11:45:23.801746: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 2 with properties: name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815 pciBusID: 0000:a6:00.0 totalMemory: 16.00GiB freeMemory: 13.32GiB 2020-05-30 11:45:23.821872: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] Device peer to peer matrix 2020-05-30 11:45:23.831541: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1051] DMA: 0 1 2 2020-05-30 11:45:23.839970: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1061] 0: Y N N 2020-05-30 11:45:23.846304: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1061] 1: N Y N 2020-05-30 11:45:23.853917: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1061] 2: N N Y 2020-05-30 11:45:23.859826: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: Quadro RTX 5000, pci bus id: 0000:17:00.0, compute capability: 7.5) 2020-05-30 11:45:23.870241: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -> (device: 1, name: Quadro RTX 5000, pci bus id: 0000:73:00.0, compute capability: 7.5) 2020-05-30 11:45:23.877451: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -> (device: 2, name: Quadro RTX 5000, pci bus id: 0000:a6:00.0, compute capability: 7.5) jay_before_sess <tensorflow.python.client.session.InteractiveSession object at 0x000001CA5DB267B8> jay_after_sess <tensorflow.python.training.saver.Saver object at 0x000001CA5DB5FE48> Data load, 100.0% finished 20000 5000 0 100 10 0 0 100 10 0 ("this is value of "step") jay_after_step 0.534315 ("this is dice value for "step" 0") 1 ("this is value of "step")
Traceback (most recent call last): File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call return fn(*args) File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn status, run_metadata) File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape [[Node: loss_function/mul = SparseDenseCwiseMul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_function/stack, loss_function/ones_like, loss_function/ToInt64_2, loss_function/Softmax)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File ".\train.py", line 144, in
train(config_file)
File ".\train.py", line 123, in train
dice = loss.eval(feed_dict ={x:tempx, w:tempw, y:tempy})
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 570, in eval
return _eval_using_default_session(self, feed_dict, self.graph, session)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 4455, in _eval_using_default_session
return session.run(tensors, feed_dict)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
run_metadata_ptr)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
options, run_metadata)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape
[[Node: loss_function/mul = SparseDenseCwiseMul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_function/stack, loss_function/ones_like, loss_function/ToInt64_2, loss_function/Softmax)]]
Caused by op 'loss_function/mul', defined at: File ".\train.py", line 144, in
train(config_file)
File ".\train.py", line 71, in train
loss = loss_func(predicty, y, weight_map = w)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\niftynet\layer\base_layer.py", line 32, in call
return self._op(*args, *kwargs)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\template.py", line 268, in call
result = self._call_func(args, kwargs, check_for_new_variables=False)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\template.py", line 217, in _call_func
result = self._func(args, *kwargs)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\niftynet\layer\loss_segmentation.py", line 61, in layer_op
pred, ground_truth, weight_map))
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\niftynet\layer\loss_segmentation.py", line 320, in dice
one_hot prediction, reduction_axes=[0])
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\math_ops.py", line 905, in binary_op_wrapper_sparse
name=name), sp_x.dense_shape)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\ops\gen_sparse_ops.py", line 851, in sparse_dense_cwise_mul
sp_shape=sp_shape, dense=dense, name=name)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "C:\Users\Jayendra\anaconda3\envs\myEnv\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InvalidArgumentError (see above for traceback): Provided indices are out-of-bounds w.r.t. dense side with broadcasted shape [[Node: loss_function/mul = SparseDenseCwiseMul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](loss_function/stack, loss_function/ones_like, loss_function/ToInt64_2, loss_function/Softmax)]]
(myEnv) PS C:\Users\Jayendra\Desktop\brats17-master\brats17-master>