I'm not actually suggesting to do wbf or nms on masks but I think those methods should be compatible with masks for instance segmentation. I haven't tried any methods in this repository for instance segmentation yet but I think it makes sense to get more refined masks with wbf and nms.
I'm currently doing this by adding masks argument to prepare_boxes and nms_method functions.
def nms_method(boxes, scores, labels, masks=None, method=3, iou_thr=0.5, sigma=0.5, thresh=0.001, weights=None):
"""
:param boxes: list of boxes predictions from each model, each box is 4 numbers.
It has 3 dimensions (models_number, model_preds, 4)
Order of boxes: x1, y1, x2, y2. We expect float normalized coordinates [0; 1]
:param scores: list of scores for each model
:param labels: list of labels for each model
:param method: 1 - linear soft-NMS, 2 - gaussian soft-NMS, 3 - standard NMS
:param iou_thr: IoU value for boxes to be a match
:param sigma: Sigma value for SoftNMS
:param thresh: threshold for boxes to keep (important for SoftNMS)
:param weights: list of weights for each model. Default: None, which means weight == 1 for each model
:return: boxes: boxes coordinates (Order of boxes: x1, y1, x2, y2).
:return: scores: confidence scores
:return: labels: boxes labels
"""
# If weights are specified
if weights is not None:
if len(boxes) != len(weights):
print('Incorrect number of weights: {}. Must be: {}. Skip it'.format(len(weights), len(boxes)))
else:
weights = np.array(weights)
for i in range(len(weights)):
scores[i] = (np.array(scores[i]) * weights[i]) / weights.sum()
# We concatenate everything
boxes = np.concatenate(boxes)
scores = np.concatenate(scores)
labels = np.concatenate(labels)
if masks is not None:
masks = np.concatenate(masks)
# Fix coordinates and removed zero area boxes
boxes, scores, labels, masks = prepare_boxes(boxes, scores, labels, masks)
# Run NMS independently for each label
unique_labels = np.unique(labels)
final_boxes = []
final_scores = []
final_labels = []
if masks is not None:
final_masks = []
for l in unique_labels:
condition = (labels == l)
boxes_by_label = boxes[condition]
scores_by_label = scores[condition]
labels_by_label = np.array([l] * len(boxes_by_label))
if masks is not None:
masks_by_label = masks[condition]
if method != 3:
keep = cpu_soft_nms_float(boxes_by_label.copy(), scores_by_label.copy(), Nt=iou_thr, sigma=sigma, thresh=thresh, method=method)
else:
# Use faster function
keep = nms_float_fast(boxes_by_label, scores_by_label, thresh=iou_thr)
final_boxes.append(boxes_by_label[keep])
final_scores.append(scores_by_label[keep])
final_labels.append(labels_by_label[keep])
if masks is not None:
final_masks.append(masks_by_label[keep])
final_boxes = np.concatenate(final_boxes)
final_scores = np.concatenate(final_scores)
final_labels = np.concatenate(final_labels)
final_masks = np.concatenate(final_masks)
return final_boxes, final_scores, final_labels, final_masks
I'm not actually suggesting to do wbf or nms on masks but I think those methods should be compatible with masks for instance segmentation. I haven't tried any methods in this repository for instance segmentation yet but I think it makes sense to get more refined masks with wbf and nms.
I'm currently doing this by adding
masks
argument toprepare_boxes
andnms_method
functions.I know it looks like error prone with this fast work around but do you think this feature deserves a pull request? @ZFTurbo