Error while training

[dmsql0816]

change batch size into 8, and training with command CUDA_VISIBLE_DEVICES="0, 1" python tools/cityscapes/train_bisenetv2_cityscapes.py I met this errors

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: slice index 262144 of dimension 0 out of bounds. [[{{node tower_3/BiseNetV2/stage_1_segmentation_loss/strided_slice}}]] [[ConstantFoldingCtrl/miou/mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Switch_0/_336]] (1) Invalid argument: slice index 262144 of dimension 0 out of bounds. [[{{node tower_3/BiseNetV2/stage_1_segmentation_loss/strided_slice}}]] 0 successful operations. 1 derived errors ignored.

It works until epoch 80 but suddenly stop. How can I fix it? And can you tell what MIN_SAMPLE_NUMS in cityscapes_bisenetv2.yaml mean?

[MaybeShewill-CV]

@dmsql0816 The min sample counts in OHEM process. Check code https://github.com/MaybeShewill-CV/bisenetv2-tensorflow/blob/ea07cc35d33989417236394c442ca42f3f8db342/bisenet_model/bisenet_v2.py#L872-L912 .

[dmsql0816]

To fix that error, should I have to fix somewhere in bisenet_v2.py?

[MaybeShewill-CV]

@dmsql0816 No. If you not familiar with ohem you may reduce MIN_SAMPLE_NUMS in config file or disable ohem in config file.

[dmsql0816]

In paper, they use ohem, so I want to use it too. I want to proceed as you training BiseNetv2-tensorflow, just change batch size into 8, and training with 2 gpus. what should I have to do to fix that error? sorry for bothering you, and thank you for your fast reply.

[MaybeShewill-CV]

@dmsql0816 If you train the model on cityscapes datase. You need to adjust some field value in https://github.com/MaybeShewill-CV/bisenetv2-tensorflow/blob/ea07cc35d33989417236394c442ca42f3f8db342/config/cityscapes/cityscapes_bisenetv2.yaml#L48-L70 . Other section (such as OHEM in solver) do not need to be adjusted:)

[dmsql0816]

It WORKS! Thank you so much!!

[MaybeShewill-CV]

@dmsql0816 No problem:)