is there a way to make masks not to overlap?

[polinamalova0]

Search before asking

• [X] I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

Dear Ultralytics team, After training my instance segmentation model, i tried to predict the objects and obtained the masks. However, the cases where they overlay one on another are very common. Therefore my question is: is there a way to set the prediction so that the masks are not overlapping (exclusive for each object)?

Additional

No response

[glenn-jocher]

Hello,

Thank you for reaching out! To prevent instance masks from overlapping in YOLOv8, you can set the overlap_mask argument to False when using the train or predict commands. Here's an example of how to implement it during prediction:

yolo predict model=path/to/your_model.pt source=path/to/your_image.jpg overlap_mask=False

This will ensure that each mask is exclusive to its respective detected object. Let us know if you need further assistance!

[polinamalova0]

Hello,

Thank you for reaching out! To prevent instance masks from overlapping in YOLOv8, you can set the overlap_mask argument to False when using the train or predict commands. Here's an example of how to implement it during prediction:

yolo predict model=path/to/your_model.pt source=path/to/your_image.jpg overlap_mask=False

This will ensure that each mask is exclusive to its respective detected object. Let us know if you need further assistance!

@glenn-jocher My problem is that i used it both during training process:

results = model.train(data=r'path/dataset.yaml', epochs=100,  imgsz=(428,428), device = device, overlap_mask=False, verbose=True)

as well as during prediction:

model = YOLO(model_path)
results = model(img, overlap_mask=False, save=True)

but it still shows me overlapping masks.

[glenn-jocher]

@polinamalova0 hello,

Thanks for the detailed information! It seems like the overlap_mask=False should indeed prevent the masks from overlapping. If it's still not working as expected, it might be a good idea to ensure that your model and the Ultralytics YOLOv8 library are both up to date. Sometimes, such issues can be resolved with the latest updates.

If updating doesn't resolve the issue, it could be helpful to check if there's any custom modification in your model or data processing pipeline that might be influencing the mask behavior.

Please let us know if the problem persists after checking these points!

[polinamalova0]

@polinamalova0 hello,

Thanks for the detailed information! It seems like the overlap_mask=False should indeed prevent the masks from overlapping. If it's still not working as expected, it might be a good idea to ensure that your model and the Ultralytics YOLOv8 library are both up to date. Sometimes, such issues can be resolved with the latest updates.

If updating doesn't resolve the issue, it could be helpful to check if there's any custom modification in your model or data processing pipeline that might be influencing the mask behavior.

Please let us know if the problem persists after checking these points!

Unfortunately, i updated the libraries as well as re-checked all my annotations to make sure that none overlap, and the newly trained model still gives off overlapped masks as a result... is there another way that can help me with the problem?

[glenn-jocher]

Hello @polinamalova0,

Thank you for the update and for checking those aspects. It's puzzling that the issue persists despite these efforts. Here are a couple more steps we can try:

1. Debugging Output: Could you please run a few test predictions and share the output logs? Sometimes, the logs can provide insights into what might be going wrong.

2. Configuration Check: Double-check the configuration file for any overlooked settings that might affect mask generation.

3. Isolate the Issue: Try running a prediction with a minimal dataset (perhaps even synthetic data where you control overlap completely) to see if the issue is data-dependent.

4. Community Input: Consider posting this issue in the Ultralytics discussions forum. Sometimes, community members may have faced and resolved similar issues.

We're here to help you through this, so keep us posted on your progress! 🚀