search

MaybeShewill-CV / bisenetv2-tensorflow

Unofficial tensorflow implementation of real-time scene image segmentation model "BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation"
https://maybeshewill-cv.github.io/bisenetv2-tensorflow/
MIT License
224 stars 59 forks source link

InvalidArgumentError (see above for traceback): slice index 262144 of dimension 0 out of bounds. #30

Closed fanchunpeng closed 3 years ago

fanchunpeng commented 3 years ago

train loss: 7.23991, miou: 0.22505: 87%|██████████████████████████████████████████████████████████████████████████▊ | 1294/1487 [12:57<02:17, 1.40it/s]2020-11-02 16:43:36.873786: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds. 2020-11-02 16:43:36.873945: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds. 2020-11-02 16:43:36.873967: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds. 2020-11-02 16:43:36.874000: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds. 2020-11-02 16:43:36.874034: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at strided_slice_op.cc:106 : Invalid argument: slice index 262144 of dimension 0 out of bounds. train loss: 7.23991, miou: 0.22505: 87%|██████████████████████████████████████████████████████████████████████████▊ | 1294/1487 [12:57<01:55, 1.66it/s] Traceback (most recent call last): File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 262144 of dimension 0 out of bounds. [[{{node tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice}}]] [[{{node ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 42, in train_model() File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 34, in train_model worker.train() File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/trainner/cityscapes/cityscapes_bisenetv2_multi_gpu_trainner.py", line 393, in train self._loss, self._global_step File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 262144 of dimension 0 out of bounds. [[node tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice (defined at /pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py:905) ]] [[{{node ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0}}]]

Caused by op 'tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice', defined at: File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 42, in train_model() File "tools/cityscapes/train_bisenetv2_cityscapes.py", line 29, in train_model worker = multi_gpu_trainner.BiseNetV2CityScapesMultiTrainer() File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/trainner/cityscapes/cityscapes_bisenetv2_multi_gpu_trainner.py", line 153, in init is_net_first_initialized=is_network_initialized File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/trainner/cityscapes/cityscapes_bisenetv2_multi_gpu_trainner.py", line 337, in _compute_net_gradients reuse=is_net_first_initialized File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py", line 1112, in compute_loss n_min=self._ohem_min_sample_nums File "/pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py", line 905, in _compute_ohem_cross_entropy_loss ohem_cond = tf.greater(loss[n_min], ohem_thresh) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 654, in _slice_helper name=name) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 820, in strided_slice shrink_axis_mask=shrink_axis_mask) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 9356, in strided_slice shrink_axis_mask=shrink_axis_mask, name=name) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/pxsj/conda_envs/server/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): slice index 262144 of dimension 0 out of bounds. [[node tower_0/BiseNetV2/seg_head_segmentation_loss/strided_slice (defined at /pxsj/fancp/ZLC/bisenetv2-tensorflow/bisenet_model/bisenet_v2.py:905) ]] [[{{node ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0}}]]

MaybeShewill-CV commented 3 years ago

@fanchunpeng If your train your custom data make sure the MIN_SAMPLE_NUMS params is not larger than your input_tensor_height * input_tensor_width in the following cinfig file. https://github.com/MaybeShewill-CV/bisenetv2-tensorflow/blob/710db8646ceb505999b9283c0837c0b5cf67876d/config/cityscapes/cityscapes_bisenetv2.yaml#L80-L83

fanchunpeng commented 3 years ago

cityscapes data ,Can train, but there are mistakes in the training process

MaybeShewill-CV commented 3 years ago

@fanchunpeng Check your local data if there is any empty data or invalid image which can not be read:)

fanchunpeng commented 3 years ago

I'll get the data again. If there are still problems, I may need to trouble you. Thank you

MaybeShewill-CV commented 3 years ago

@fanchunpeng That problem won't happen if you train the model on cityscapes dataset. Just make sure the local images are invalid and can be correctly read:)

fanchunpeng commented 3 years ago

train loss: 10.97951, miou: 0.36470: Thank you. After reorganizing the data, everything is normal. How long did you train? I found that Miou grew slowly.

MaybeShewill-CV commented 3 years ago

@fanchunpeng I do not remember the exact time consuming:)