Training for rotated bounding boxes

[frankbrewer]

I was wondering whether anyone added a parameter for the angle of rotation and tried training to get outputs like example-c in the image below.

How hard do you think it would be to accomplish this and what is your wild guess about what the networks precision would be for such an application?

[AlexeyAB]

First we have to find good ones:

1. labeling tools which support rotated bboxes
2. dataset with rotated bboxes (also script that converts these bboxes to the yolo format, and script that checks accuracy (mAP) for rotated bboxes)

Can you provide links to such tools and datasets?

Then I can add rotated-bbox feature to the [yolo] layer, it is not very difficult, but it will take time, as in many places hardcoded number of coordinates.

[clw5180]

First we have to find good ones:

1. labeling tools which support rotated bboxes
2. dataset with rotated bboxes (also script that converts these bboxes to the yolo format, and script that checks accuracy (mAP) for rotated bboxes)

Can you provide links to such tools and datasets?

Then I can add rotated-bbox feature to the [yolo] layer, it is not very difficult, but it will take time, as in many places hardcoded number of coordinates.

Hi @AlexeyAB ! Hope that you can add rotated-bbox feature!

1. This tool can help ------ https://github.com/chinakook/labelImg2
2. ICDAR 2015, or use the tool above to label some picture......

[clw5180]

And dear @AlexeyAB , Can you give some advice on how to modify the structure of yolov3 in the field of textile defect detection（object has little semantic information), maybe reduce the conv layers can help?

And some defects are long and thin with a angle of ~45, so normal bbox can't work, because of large area of background in bbox. I will try to add rotated-bbox feature in the PyTorch version of yolov3 later.

[AlexeyAB]

There may be even more suitable semantic/instance segmentation rather than rotate-bboxes. This isn't implemented yet: https://github.com/AlexeyAB/darknet/issues/3048

And some defects are long and thin with a angle of ~45, so normal bbox can't work, because of large area of background in bbox.

It can't work for what reason:

1. does yolo incorrectly detect such bboxes?
2. or because you need more accurate localization of the object - you need rotates bboxes?

[clw5180]

There may be even more suitable semantic/instance segmentation rather than rotate-bboxes. This isn't implemented yet: #3048

And some defects are long and thin with a angle of ~45, so normal bbox can't work, because of large area of background in bbox.

It can't work for what reason:

1. does yolo incorrectly detect such bboxes?
2. or because you need more accurate localization of the object - you need rotates bboxes?

can't detect such bboxes, mAP is always 0 for this class

[AlexeyAB]

Can you show:

1. example of training image with bbox
2. example of detected bbox
3. screenshot of mAP calculation

[clw5180]

There may be even more suitable semantic/instance segmentation rather than rotate-bboxes. This isn't implemented yet: #3048

And some defects are long and thin with a angle of ~45, so normal bbox can't work, because of large area of background in bbox.

It can't work for what reason:

1. does yolo incorrectly detect such bboxes?
2. or because you need more accurate localization of the object - you need rotates bboxes?

And I try to break the long defect into small segments of bboxes, it worked.

[AlexeyAB]

And I try to break the long defect into small segments of bboxes, it worked.

Yes, you can try to use this way.

What cfg-file did you use, is it yolov3-spp.cfg?

[clw5180]

And I try to break the long defect into small segments of bboxes, it worked.

Yes, you can try to use this way.

What cfg-file did you use, is it yolov3-spp.cfg?

yes, and I use the version of ultralytics-yolov3, I don't know whether it is robust.

[dexception]

For reference: https://github.com/DetectionTeamUCAS/RetinaNet_Tensorflow_Rotation

[vsemecky]

1. labeling tools which support rotated bboxes

As a labeling tool you can use Hasty: https://hasty.ai/

In fact, it doesn't support rotated bboxes, but it supports any polygon, which can be used for rotated bboxes, but also for more complex polygons. Look at this:

[sdimantsd]

Hi @AlexeyAB. Anything new with this feature?

[sdimantsd]

Hi @AlexeyAB again. Im try to modify your code to get an angle of the bounding box. but i can't overfitting my dataset (only 15 imgs). Can you help me with this?

[AlexeyAB]

1. How did you check that you can't overfit your model?
2. You should disable data augmentation and batch normalization.

[sdimantsd]

Hop i did it correctly...

• I change the output of YOLO layer from 1+4 to 1+5 in yolo_layer.c (The 5th feature is the angle from x axias. (It value is 0-1. 0 is 0 degrees and 1 is 90 degrees)).
• in delta_yolo_box() i add: delta[index + 4 * stride] += scale * (t_angle - x[index + 4 * stride]) * iou_normalizer;
• I saw that you divide X and Y by the width/height of the last layer and do a log to the bounding box width / height. I didn't do anything like that for the angle, it is good?
• About the last note, when the angle is greater than 45 degrees, the width becomes the height and the height laterally. Can't it cause problems?
• Do i need to change the loss calculation?

About your questions:

1. I changed the label file instead of 5 properties per object (class, center position, height, width) to 6 properties (I added the angle). I changed the cfg file and set filters = (classes + 6) * 3. I trained with tiny yolo and checked after two days. There were a few images he was able to detect but with low probability (0.23) and with no angle accuracy. And there were some who could not detect an object at all.
2. I didn't do it :-(

Thanks! If I need to do something else, I'd love to write to me.

[AlexeyAB]

Im try to modify your code to get an angle of the bounding box. but i can't overfitting my dataset (only 15 imgs). Can you help me with this?

Overfitting is a bad process that interferes with learning. Do you really want this?: https://en.wikipedia.org/wiki/Overfitting

There were a few images he was able to detect but with low probability (0.23) and with no angle accuracy.

How did you check this?

What activation do you use for angle? https://github.com/AlexeyAB/darknet/blob/d51d89053afc4b7f50a30ace7b2fcf1b2ddd7598/src/yolo_layer.c#L769-L777

If you added LOGISTIC activation for Angle, then you should normalize angle for range [0 - 1]

[sdimantsd]

Overfitting is a bad process that interferes with learning. Do you really want this?: https://en.wikipedia.org/wiki/Overfitting

I know it's a bad idea, i just what to check if i can get good angle accuracy.

How did you check this?

I saw the network output (in debug mode).

What activation do you use for angle?

Oops... forgot it...

So of course I have to normalize the angle (LOGISTIC is what i need?) And disable data augmentation and batch normalization.

Anything else? I will try to do it anyway for me. Do you think you'll be able to get to that soon?

Thank you!

[AlexeyAB]

So of course I have to normalize the angle (LOGISTIC is what i need?)

Yes. Or you can try to use sin() ro cos() https://github.com/AlexeyAB/darknet/issues/4360#issuecomment-565719927

Modify this function for loading angles too: https://github.com/AlexeyAB/darknet/blob/d51d89053afc4b7f50a30ace7b2fcf1b2ddd7598/src/data.c#L355-L444

And disable data augmentation and batch normalization.

Comment all that is related to data augmentation in cfg file.

Dont disable batch normalization.

[sdimantsd]

Thx! So just doing so (and what i write before) shuld let me know the angle?

[sdimantsd]

@AlexeyAB One more thing. What i need to change in the code? I can see some numbers im shure i need to change, but in not shure i get all of them, especially in delta_yolo_box() and all the thin around it. Sorry to interupt you, and thanks!

[sctrueew]

@sdimantsd Hi,

Could you share the code if it's completed?

[sdimantsd]

@zpmmehrdad Hi, unfortunately I couldn't do it. I'm also waiting for @AlexeyAB to update if he finds time to do.

[sctrueew]

@AlexeyAB Hi,

I have some classes that need to be labeled as rotating rectangles or polygonal, Like Mask-RCNN. Can we have this feature on this repo?

Current situation:

We need:

Thanks

[AIFAN-Lab]

Hop i did it correctly...

• I change the output of YOLO layer from 1+4 to 1+5 in yolo_layer.c (The 5th feature is the angle from x axias. (It value is 0-1. 0 is 0 degrees and 1 is 90 degrees)).
• in delta_yolo_box() i add: delta[index + 4 * stride] += scale * (t_angle - x[index + 4 * stride]) * iou_normalizer;
• I saw that you divide X and Y by the width/height of the last layer and do a log to the bounding box width / height. I didn't do anything like that for the angle, it is good?
• About the last note, when the angle is greater than 45 degrees, the width becomes the height and the height laterally. Can't it cause problems?
• Do i need to change the loss calculation?

About your questions:

1. I changed the label file instead of 5 properties per object (class, center position, height, width) to 6 properties (I added the angle). I changed the cfg file and set filters = (classes + 6) * 3. I trained with tiny yolo and checked after two days. There were a few images he was able to detect but with low probability (0.23) and with no angle accuracy. And there were some who could not detect an object at all.
2. I didn't do it :-(

Thanks! If I need to do something else, I'd love to write to me.

Hello, I just modify the code. But it dosn't work as prediced. Can you share me some code about how to process [class,x,y,w,h,angle]?

[sdimantsd]

Hi, Although I changed the code everywhere I thought, it doesn't give any logical results even on 5 pictures. I guess I didn't change all the places I needed. I'm waiting for @AlexeyAB to do it.

If you succeed, I would love for you to post here. Thanks

[sdimantsd]

@AlexeyAB Hi,

I have some classes that need to be labeled as rotating rectangles or polygonal, Like Mask-RCNN. Can we have this feature on this repo?

Current situation:

We need:

Thanks

It's not just a rotated square, in case you have a 4-corner polygon here

[ineffable-yp]

friends, does anyone have a executed repo about this issue? I also need to train for rotated bounding boxes, I find it hard to complete for me.

[sdimantsd]

Hi, I try to, but I didn't success :-(