david8862 / keras-YOLOv3-model-set

end-to-end YOLOv4/v3/v2 object detection pipeline, implemented on tf.keras with different technologies
MIT License
640 stars 222 forks source link

Training stops mid-way during first epoch #193

Closed ZiyueWangUoB closed 3 years ago

ZiyueWangUoB commented 3 years ago

I used the following code to run train.py

python train.py --model_type tiny_yolo3_mobilenet --anchors_path configs/tiny_yolo3_anchors.txt --annotation_file brainwash/train.txt --val_annotation_file brainwash/val.txt --classes_path configs/coco_classes.txt --batch_size 16

and the error I got was ` Traceback (most recent call last):

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 249, in call ret = func(*args)

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 645, in wrapper return func(*args, **kwargs)

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 961, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 837, in wrapped_generator for data in generator_fn():

File "/home/ubuntu/Documents/yolov4/keras-YOLOv3-model-set/yolo3/data.py", line 269, in yolo3_data_generator image, box = get_ground_truth_data(annotation_lines[i], input_shape, augment=True)

File "/home/ubuntu/Documents/yolov4/keras-YOLOv3-model-set/yolo3/data.py", line 73, in get_ground_truth_data boxes = reshape_boxes(boxes, src_shape=image_size, target_shape=model_input_size, padding_shape=padding_size, offset=padding_offset, horizontal_flip=horizontal_flip, vertical_flip=vertical_flip)

File "/home/ubuntu/Documents/yolov4/keras-YOLOv3-model-set/common/data_utils.py", line 144, in reshape_boxes boxes[:, [0,2]] = boxes[:, [0,2]]*padding_w/src_w + dx

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

[[{{node PyFunc}}]] [[IteratorGetNext]] (1) Unknown: IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed Traceback (most recent call last):

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 249, in call ret = func(*args)

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 645, in wrapper return func(*args, **kwargs)

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 961, in generator_py_func values = next(generator_state.get_iterator(iterator_id))

File "/home/ubuntu/miniconda3/envs/tinyYolo/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 837, in wrapped_generator for data in generator_fn():

File "/home/ubuntu/Documents/yolov4/keras-YOLOv3-model-set/yolo3/data.py", line 269, in yolo3_data_generator image, box = get_ground_truth_data(annotation_lines[i], input_shape, augment=True)

File "/home/ubuntu/Documents/yolov4/keras-YOLOv3-model-set/yolo3/data.py", line 73, in get_ground_truth_data boxes = reshape_boxes(boxes, src_shape=image_size, target_shape=model_input_size, padding_shape=padding_size, offset=padding_offset, horizontal_flip=horizontal_flip, vertical_flip=vertical_flip)

File "/home/ubuntu/Documents/yolov4/keras-YOLOv3-model-set/common/data_utils.py", line 144, in reshape_boxes boxes[:, [0,2]] = boxes[:, [0,2]]*padding_w/src_w + dx

IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

[[{{node PyFunc}}]] [[IteratorGetNext]] [[IteratorGetNext/_8]] 0 successful operations. 0 derived errors ignored. [Op:__inference_train_function_13440]

Function call stack: train_function -> train_function `

I'm using the brainwash dataset and have converted the labels to the same file structure as instructed. Could this be a tensorflow problem? (Using tensorflow 2.5.0 with CUDA 11.2)

ZiyueWangUoB commented 3 years ago

Found the source of the error, there was an issue in the training data where the bounding box was missing the ymax value. Resolved.