Polygons instead of rectangular bounding box

[BS-98]

Hi,

is it possible to put polygons as an input instead of rectangular bounding boxes? I am interested in semantic segmentation and I would like provide the data in form of polygons.

I would be grateful for any help.

[andreaalf97]

+1

[alcinos]

Hi @BS-98 Thank you for your interest in DETR. It should be rather straight-forward to extend DETR to arbitrary polygons, as long as the number of vertex is constant:

• In the box head of the model, instead of computing 4 scalars that correspond to the (x,y) coordinates of the center and the height and width of the box, you can, for example, return 2*N scalars that correspond to the coordinates of the N vertex
• In the criterion, you need to modify the way the loss between two polygons is computed. I guess the L1 loss that we have is fine to use more or less as-is, but if you want to use something like the generalized IoU that we use, you'll need to reformulate it to handle arbitrary polygons (it involves finding unions and intersection of polygons, which might be a pain to compute).
• Similarly to the criterion, you need to modify the matching cost so that it computes the same thing as the loss.

That being said, if your end goal is semantic segmentation, I recommend that you look at the panoptic segmentation approach that we propose. Note that panoptic segmentation is a mix of semantic segmentation and instance segmentation, so technically it is a super-set of what you want to do.

Feel free to ask if anything is unclear.

[TG-Unit]

Hi @BS-98

Have you been successful with this approach? I am working on a similar task and would be interested if you were able to solve this.

BR

[yellowjs0304]

I wanna make same result like rect to polygon (before : x, y, w, h -> after : x1, y1, x2, y2, x3, y3, x4, y4). Do you have any good sample like this? i already modify "self.bbox_embed" into MLP(hidden_dim, hidden_dim, 4, 3) to MLP(hidden_dim, hidden_dim, 8, 3)

Also, I think i need to change the "loss_boxes" but, I don't know how to modify it... @alcinos @BS-98