Segmentation is wavy and has an offset when I train different segmentation models with YOLOv8

[SteFehr]

Search before asking

• [X] I have searched the YOLOv8 issues and discussions and found no similar questions.

Question

I want to perform instance segmentation with a YOLOv8 network. I tried to train the model with various parameters which are related to segmentation tasks (mask_ratio, retina_masks, ... ), but the output of the segmentation model is always wavy and has a small offset (see the picture below).

Offset: Lower and right object edge -> also background is partly segmented Upper and left object edge -> object is not fully segmented

I am training on a mixed dataset of synthetic and real images, which are annotated precisly.

Anybody else who was experienced this behaviour?

Additional

No response

[github-actions[bot]]

👋 Hello @SteFehr, thank you for your interest in YOLOv8 🚀! We recommend a visit to the YOLOv8 Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Install

Pip install the ultralytics package including all requirements in a Python>=3.7 environment with PyTorch>=1.7.

pip install ultralytics

Environments

YOLOv8 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLOv8 Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

Hello @SteFehr! Thank you for your question.

It sounds like you are experiencing segmentation errors in your YOLOv8 network. Based on what you have described, one possibility for what you are encountering could be related to a misalignment between your ground truth masks and the predicted masks.

You mentioned you are training on a mixed dataset of synthetic and real images with precise annotation, which is great. However, one thing you can investigate further is to ensure that the dimensions of the ground truth masks match the output dimensions of your YOLOv8 network. Another possibility is to experiment with different augmentation techniques which could improve the alignment between input images and segmentation masks.

I hope this helps! Let us know if you have any further questions or if there is anything more we can do to help.

[SteFehr]

Hello @glenn-jocher,

Thank you for the reply. I don't exactly know what you are meaning with "ensure that the dimensions of the ground truth masks match the output dimensions of your YOLOv8 network". Could you please provide more details?

Furthermore I tried various augmentation techniques, but the issue is still there.

[SteFehr]

I think I understand the problem now. The input images have dimensions of 1792x1233 pixels. But I train the model with the parameter image size 640, so the images as well as the labels are reduced to this size. When predicting, the segmentation is applied to the image with size 640 and then scaled up to the original image size. This is probably the reason for these "waves". I also set the retina_masks parameter to true and it looks like the peaks of these "waves" are smoothed out a bit (this is what causes the supposed offset).

With imgsz parameter set to 1280 I get better results as far as segmentation in general is concerned, but for objects that appear very large in the image, not the whole area of the object is detected (see image below). Do you have any explanation for this? It does not happen when I train with imgsz parameter 640 as an example.

[glenn-jocher]

Thank you for providing further information about your issue @SteFehr. Yes, you are correct that the segmentation is applied to the image with size 640 and then scaled up to the original image size, which can cause the "waves" that you see in the output segmentation map.

Regarding the issue you are seeing with objects appearing very large in the image not being fully detected, this could be related to the max stride of the YOLOv8 network. When objects are very large in the image, the receptive field of the YOLOv8 network may not cover the entire object, and the output segmentation may not fully capture the object. To solve this, one approach is to decrease the max stride of the network so that the output has a higher resolution. However, a smaller max stride can slow down the network and require more memory, so there is a tradeoff to consider.

I hope this helps! Let me know if you have any more questions or if there's anything else I can help with.

[SteFehr]

Hey @glenn-jocher, thank you for the tip! How can i change the max stride of the network? I only see a "vid_stride" parameter in the args.yaml file.

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[leo-q8]

@SteFehr @glenn-jocher Hi~，I also encountered this problem that the predicted segmentation has a small offset, like this:

The input image have 1392x1040 pixels. And I train and predict at 1280 imgsz. I also try to use 1280 as predicted imgsz, but there still has a offset. Is this problem caused by model or the postprocess? Could you give me some advice?

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[farhanrw]

Facing the same problem. @SteFehr @xueqing888 did you manage to solve this?

[Boatsure]

Same problem here！The mAP value is high and good, I'm checking the masks visualization postprcocess code.

[glenn-jocher]

Hello! It sounds like you're making good progress with your model's performance. Since your mAP values are satisfactory but you’re still facing issues with masks visualization, indeed double-checking the post-processing steps for visualizing the segmentation masks could be helpful. Ensure that the mask resizing and overlay steps onto the original images are correctly handled. Sometimes mismatches in these steps can cause discrepancies in the visual output. 👍

If you need any specific code snippets or further assistance, feel free to ask!

[lixinghuaxd]

Hello, I also encountered the same problem. Have you solved the wavy line issue

[pderrenger]

Hello! The wavy line issue in segmentation can often be attributed to image resizing and post-processing steps. Ensure you're using the latest version of the Ultralytics package to rule out any bugs that may have been fixed in recent updates. Additionally, double-check your mask resizing and overlay processes to ensure they align correctly with the original image dimensions. If the problem persists, feel free to share more details for further assistance.

[lixinghuaxd]

Hello! The wavy line issue in segmentation can often be attributed to image resizing and post-processing steps. Ensure you're using the latest version of the Ultralytics package to rule out any bugs that may have been fixed in recent updates. Additionally, double-check your mask resizing and overlay processes to ensure they align correctly with the original image dimensions. If the problem persists, feel free to share more details for further assistance.

Hello, this is the correct mask for the training data. I saved the 160 * 160 mask directly output by the model, which also shows wavy lines train_batch0.jpg

[pderrenger]

@lixinghuaxd thank you for sharing the additional details. Given that the wavy lines appear directly in the model's output, this issue might be related to the model's architecture or training parameters. Please ensure you are using the latest version of the Ultralytics package. If the issue persists, consider experimenting with different hyperparameters or data augmentation techniques to improve the mask quality. If you continue to face this problem, providing more details about your training setup and parameters could help diagnose the issue further.

[sctrueew]

Hi, I have the same problem. my training data is correct but I trained a model with yolov8l with retina_masks, and the output is wavy.

[glenn-jocher]

Hi, thanks for reaching out. This issue might be related to mask resolution or training settings. Try adjusting the imgsz parameter or experimenting with different augmentation techniques to see if it improves the results.