[CoreML] Auto labeling multiple (connected) "ghost" polygon issue

[Denizzje]

Greetings,

I have been using the auto label feature with great succes the past few weeks. However, I do notice a rather strange habit of the auto labeling tool. See the picture below:

Sometimes, the auto labeling tool randomly connects two polygons. I get alot of points at a random area in the picture. In the picture, the form above is correct, but it shouldnt draw on the damaged part of the glass. Sometimes it just happens in a random area. In the dataset I am labelling, it seems about 1/3 of the pictures have this issue.

Now I have two options to solve this:

• Delete all the points of the ghost polygon, one by one
• Delete the auto label polygon, go to create box, redo the labeling. Most of the times, it will still draw the incorrect "ghost" polygon(s) apart from the correct ones, but then it does not connect them together so I can easily delete them with "seperate polygon" function.

Both, and especially the first method can be quite time consuming. Is it perhaps possible, that one can select a group of those individual polygon points, right click and delete those points? I think that would be the best solution in this case.

The model I am using is a custom YoloV8 Segmentation model of the L size to be complete. But I think we cannot find the root cause here. It is perhaps because of how YoloV8 Segmentation works, and/or the conversion to CoreML causing these ghost polygons. I do not find the ghost polygons to be a big issue, if they can be easily deleted.

[ryouchinsa]

Thanks for writing the issue. We will implement the erase mode for polygon. In the erase mode, when you finish drawing a polygon, points of the currently selected polygon which are inside the erase polygon will be removed.

By the way, if you auto label your YOLOv8 segmentation model in the PASCAL VOC xml mode, the pixels mask is saved. In the YOLO text mode, we are converting the pixels mask to polygons, and merge the polygons into a merged polygon. So that multiple polygons are connected with thin lines.

[Denizzje]

Hi @ryouchinsa , thank you once again. The erase polygon sounds like a good idea to combat this issue and lower label time in this more difficult dataset where the YoloV8 model is a bit too happy with creating masks / polygons.

Do you recommend that when using YoloV8, I should (auto)label in the Pascal VOC mode so that I will not encounter the merged polygons, and then later when I have finished I use the "export Yolo txt files" function. Instead of using the auto label function in Yolo TXT mode? Am I understanding correctly?

Irregardless, aside from the the merged connected polygon issue, the erase mode for polygon will be very useful because sometimes I have alot of overlap in to be labeled objects and it gets a bit messy with the polygons at the place of overlap and Yolo models can struggle alot with objects overlapping. Being able to easily fix that with the erase polygon will be very useful and I look forward to it!

[ryouchinsa]

We improved so that toggling the erase mode, you can use the polygon tool for erasing multiple points of the selected polygon. https://rectlabel.com/edit#erase

1. Change to create polygon mode.
2. Use a shortcut to toggle the erase mode ON.
3. Draw a polygon with a dashed line and enter key to remove points which are inside the polygon.

To understand how to convert the mask image to polygons, PASCAL VOC xml format is useful. But to speed up your labeling work for YOLOv8, we recommend to use the YOLO text format.

Please let us know your feedback.

[Denizzje]

Sorry I forgot to respond. I can confirm that this works very well and it really speed up my labeling. This is a great addition to the tool.

Sometimes the erase mode didnt seem to get triggered properly when I clicked the button. But that might be because I am not used to it properly or I am doing something wrong. I hope to have some time to label again this week to give it another try.

Edit: Currently I think this was caused because I was working remotely and my mouse does not register properly sometimes with the remote desktop software. I will update the feedback when I get into the office to label again.

[Denizzje]

Hi @ryouchinsa , it seems the issue I mentioned aboout the erase mode not triggering correctly was indeed caused by me working remotely over Anydesk. Infact there were some other issues such as not being able to set key shortcuts remtoely.

I am closing this issue and I thank you again for your brilliant work. It saved me a lot of time already.

As I am new to MacOS, if you know better remote software than Anydesk please advise me by the way, I work remote 60-80% of the time and I would like to label remotely 😆 .

[ryouchinsa]

Currently we provide 2 options to erase the polygon points such as "Erase points" and "Erase mask". The erase polygon panel is shown when you are using the erase mode. We hope this would help for your use-case.

About the remote software such as Anydesk, when we find out better software, we will let you know.

[Denizzje]

Hi @ryouchinsa , thank you for another update and all the work you put in for this function. I have saved a great deal of time already with the erase function. I will try the erase mask function as well next week and give you feedback.

Am I correct that currently, there is no keyboard shortcut to toggle between erase and create mode? If I am correct, is there a possibility that it can be added? It is not a big deal, just good to have if possible :).

[ryouchinsa]

Yes, you can customize the toggling erase mode hotkey on the setting dialog.