poor mask performance when overlapping box

[liminn]

@dbolya Hi, I use yolact++ to perform handwriting circle instance segmentation. After training, my custom benchmark metric is:

• box_mAP_0.5: 97.91
• mask_mAP_0.5:93.11

When the circle is isolated, the prediction results is right. But when prediction box is overlapping, the prediction instance mask performance is poor, for example:

Compared with the original config, I only modify:

'augment_expand': False, #True, 
'discard_mask_area': -1, #5*5,

By the way, I use the RLE to encode the instance mask, so I ensure my instance mask training label is right.

[Kasdision]

Hi，when you try your own data, have you changed the pred_scales and pred_aspect_ratios for the box or anything else? Cause it looks like something wrong in the config file.