Issues applied to other multi-segmentation and detection tasks

[ZhanYeWei]

First of all thank you very much for your outstanding work! I would like to apply the model in other areas of research, in my scenario the seg task needs to output classification results (i.e. not just one class of 10: drivable, but multiple different classes that need to be segmented). I tried to modify database.yaml and model.yaml according to the documentation, but still have some doubts: 1) If I change database.yaml to: tnc: 5 nc_list: [1,4,0] map: [None,None,None] is this possible? 2) What should I change in the detection head output section of model.yaml 3) If I only need a segmentation detection head, can I just remove a segmentation module in model.yaml? Or I need to do something more? Thanks for your help and answer!

[JiayuanWang-JW]

Thanks for your interest.

1. No, you can not do that in this code. Our code can not support your idea. You need to change some related code to implement it. Our code only supports one class in one segmentation head. However, it will be easy to implement your idea based on our code. You only need to change some related parts to achieve it. Or maybe you can use multi-segmentation heads to easily achieve it. In this way, I think you needn't change the code.
2. You can change the backbone and neck structure in model.yaml. For head parts, you are required to change the head file.
3. No, you need to change some code to achieve one segmentation head to implement multi-class segmentation.

I am looking for receiving your findings and experiment results.

[kiming-ng]

Thanks for your advice.

Based on your code, we simply added two segmentation heads to detect one class and segment four classes. We encountered the following issue when applying Mosaic augmentation:

We further investigate this issue (see figure below) and find that this problem was due to the length of our labels being 5 (one for detection, four for segmentation). When adding the mix_label parameter based on indexes(len(indexes)=3, line 91) required for Mosaic, we couldn't assign mix_label to the last two elements in the labels (line 116), leading to the above error. Since Mosaic augmentation is typically in the form of 2x2 or 3x3, we would like to know how to use Mosaic augmentation in our setup (len(labels)=5).

Thanks for your help and answer!

[ZhanYeWei]

I'll add that when I try to turn off Mosaic augmentation it seems to run through, but the results are completely incorrect, all 4 tasks except the detection task seem to be functioning properly, here are some of the results that came up, I hope you can provide some help or advice on this issue， thanks again for your reply and help!

[JiayuanWang-JW]

I'll add that when I try to turn off Mosaic augmentation it seems to run through, but the results are completely incorrect, all 4 tasks except the detection task seem to be functioning properly, here are some of the results that came up, I hope you can provide some help or advice on this issue， thanks again for your reply and help!

This file's results are not correct. You need to focus on the output from each epoch if you set val=True during training. please see your terminal output. That one is the most accurate. I didn't correct "result.csv" results.

[ZhanYeWei]

I'm very sorry to bother you again, I realized that I had some problems with my training setup before, when I added two segment headers the system reported the following error, does it mean that I need to make more changes to the loss.py, or is it just a matter of modifying it for this one spot, thank you for your reply and help!

[JiayuanWang-JW]

I'm very sorry to bother you again, I realized that I had some problems with my training setup before, when I added two segment headers the system reported the following error, does it mean that I need to make more changes to the loss.py, or is it just a matter of modifying it for this one spot, thank you for your reply and help!

Can you show me the error information? I can not see any error information in your figure.

[ZhanYeWei]

Sorry I missed your reply before, a few days ago we successfully solved the above problem and trained a model successfully, but I feel that there are some problems in performing the prediction task, the mask shape of the segment task is roughly what I thought, but the pixel points are very sparse (as shown in the picture below) and the segment task can't show the labels. Do you know what causes this? Thanks for your reply!

[yinheyanxian]

Sorry I missed your reply before, a few days ago we successfully solved the above problem and trained a model successfully, but I feel that there are some problems in performing the prediction task, the mask shape of the segment task is roughly what I thought, but the pixel points are very sparse (as shown in the picture below) and the segment task can't show the labels. Do you know what causes this? Thanks for your reply!

Hello, I have also removed a segmentation head, my code only performs one detection and one segmentation task, it can be well trained and tested. However, when predict.py was executed, there was a mismatch between the dimensions and 720. I carefully checked my code to ensure that the picture size was the same during training and prediction, but there was still a mismatch between the dimensions and 720. Do you have this problem?