search

xuebinqin / BASNet

Code for CVPR 2019 paper. BASNet: Boundary-Aware Salient Object Detection
MIT License
1.39k stars 251 forks source link

Evaluation Metric #20

Closed ftlong6666 closed 4 years ago

ftlong6666 commented 5 years ago

Hello @NathanUA thank you for the great work and code. I have several problems as follows.

  1. I am trying to reproduce the evaluation metric. Do you have a code for evaluating the results?
  2. For the precision and recall did you use the same threshold for prediction and ground truth?, so what happens if the threshold is 0 which means all of the pixels will be white and means the true positive will be high since all pixel in prediction and ground truth becomes white?, and also what happen if the threshold is 1?
  3. Could you explain the relaxed F-measure?, I am still confused and trying to figuring out is it like these steps?

-Thank you-

xuebinqin commented 5 years ago

On Sat, Oct 12, 2019 at 1:18 PM ftlong6666 notifications@github.com wrote:

Hello @NathanUA https://github.com/NathanUA thank you for the great work and code. I have several problems as follows.

  1. I am trying to reproduce the evaluation metric. Do you have a code for evaluating the results?

Yes, we reimplemented the evaluation. But currently its a bit messy. I need to reorganize them and can't upload them right now.

  1. For the precision and recall did you use the same threshold for prediction and ground truth?, so what happens if the threshold is 0 which means all of the pixels will be white and means the true positive will be high since all pixel in prediction and ground truth becomes white?, and also what happen if the threshold is 1?

We use the mybins = np.arange(0,256) to get the thresholds. The computation method are adapted from https://github.com/ArcherFMY/sal_eval_toolbox.

  1. Could you explain the relaxed F-measure?, I am still confused and trying to figuring out is it like these steps?

    • Binarize prediction (pred) and ground truth (gt) with threshold = 0.5
    • Do an erode operation in both pred and gt, how to do it? what is your kernel size? is it 3x3 as stated in here https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_morphological_ops/py_morphological_ops.html
    • Do XOR operation in eroded and non-eroded version which means the different pixels will be white and the same pixels will be black (it will be like edge only).
    • The precision and recall will be computed in predXOR and gtXOR however I am still confused about slack p where it set to be 3 pixels. then the relaxed F measure is similar computation with the original F measure.

Follow is the code for boundary extraction from binarized mask and the error computation: def boundary_extraction(mask):

(h,w) = mask.shape
mask_pad = np.zeros((h+10,w+10))
mask_pad[5:h+5,5:w+5] = mask

mask_pad_erd = erosion(mask_pad,disk(1))
mask_pad_edge = np.logical_xor(mask_pad, mask_pad_erd)

return mask_pad_edge[5:h+5,5:w+5]

-Thank you-

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NathanUA/BASNet/issues/20?email_source=notifications&email_token=ADSGORNSJAJMJR7KLY3PF6TQOIPIRA5CNFSM4JAEBOR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HRMM7DQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSGORLTGS4W4WZDOIMLWK3QOIPIRANCNFSM4JAEBORQ .

def compute_align_error(gt,pd):

gt_bw = np.zeros(gt.shape)
pd_bw = np.zeros(pd.shape)

#binaries gt
gte = 127 #2*np.mean(gte)
gt_bw[gt>gte]=1
gt_bw[gt<=gte]=0

#binaries pd
pde = 127
pd_bw[pd>pde]=1
pd_bw[pd<=pde]=0

gt_edge = boundary_extraction(gt_bw)
pd_edge = boundary_extraction(pd_bw)

gt_dist = distance_transform_edt(np.logical_not(gt_edge))
pd_dist = distance_transform_edt(np.logical_not(pd_edge))

buffer = 3
#precision
pd_edge_buffer = np.zeros(pd_edge.shape)
pd_edge_buffer = pd_edge.copy()
try:
    pd_edge_buffer[gt_dist>buffer] = 0
except:
    return 0.0, 0.0, 0.0

#recall
gt_edge_buffer = np.zeros(gt_edge.shape)
gt_edge_buffer = gt_edge.copy()
try:
    gt_edge_buffer[pd_dist>buffer] = 0
except:
    return 0.0, 0.0, 0.0

precision_edge =

np.sum(pd_edge_buffer).astype(np.float)/(np.sum(pd_edge).astype(np.float)+1e-8)

recall_edge =

np.sum(gt_edge_buffer).astype(np.float)/(np.sum(gt_edge).astype(np.float)+1e-8)

f1_edge =

(1+0.3)precision_edgerecall_edge/(0.3*precision_edge+recall_edge+1e-8)

return precision_edge, recall_edge, f1_edge #meanAll

-- Xuebin Qin PhD Candidate Department of Computing Science University of Alberta, Edmonton, AB, Canada Homepage:https://webdocs.cs.ualberta.ca/~xuebin/

Vozf commented 4 years ago

@NathanUA There seemed to be unfinished plans to release the evaluation code. Maybe you will consider doing it now that u-2-net is released?

CeciliaPYY commented 3 years ago

Hello, I used the code you offered to calculate the f_relax_measure between my gt and pred mask, I doubt the way that why the disk size is set to 1, cause under that occasion, the erosion may always be zero?? Is there something I misunderstood???

def boundary_extraction(mask, k_size=1):
    """
    对结果进行腐蚀，进而得到边缘
    """

    (h, w) = mask.shape
    mask_pad = np.zeros((h+10, w+10))
    mask_pad[5:h+5, 5:w+5] = mask

    mask_pad_erd = ndimage.grey_erosion(mask_pad, size=(k_size, k_size))
    mask_pad_edge = np.logical_xor(mask_pad, mask_pad_erd)

    return mask_pad_edge[5:h+5, 5:w+5]

pred = cv2.imread(
    '/mnt/work/yuyanpeng/code/personal/U-2-Net/test_data/u2net_results/ILSVRC2012_test_00052503.png', 0)

gt = cv2.imread(
    '/mnt/work/yuyanpeng/code/personal/PoolNet/data/DUTS-TE/DUTS-TE-Mask/ILSVRC2012_test_00052503.png', 0)

gt[gt <= 2*np.mean(gt)] = 0
gt[gt > 2*np.mean(gt)] = 1

pred[pred <= 127] = 0
pred[pred > 127] = 1

pred_edge = boundary_extraction(pred)
gt_edge = boundary_extraction(gt)

# distance_transform_edt 用于距离转换，计算图像中非零点到最近背景点（即0）的距离
# 这里先将背景和前景互换，即背景变成1，前景变成0，则变为计算背景中距离最近边缘的距离
pred_dist = distance_transform_edt(np.logical_not(pred_edge))
gt_dist = distance_transform_edt(np.logical_not(gt_edge))

# The relaxed boundary recall (relaxRecallb) measures the fraction of ground truth boundary pixels 
# that are within ρ pixels of predicted boundary pixels. Here ρ sets to 3.
buffer = 3
pred_edge_buffer = np.zeros_like(pred_edge)
pred_edge_buffer = pred_edge.copy()

# precision = TP / 总数
pred_edge_buffer[gt_dist > buffer] = 0  # buffer 之外的点（即距离最近的边缘超过3个像素的点不考虑在内）
precision_edge = np.sum(pred_edge_buffer).astype(np.float)/(np.sum(pred_edge).astype(np.float) + 1e-8)
precision_edge

# recall = TP / (TP + FN)
gt_edge_buffer = np.zeros_like(gt_edge)
gt_edge_buffer = gt_edge.copy()
gt_edge_buffer[pred_dist > buffer] = 0  # 
recall_edge = np.sum(gt_edge_buffer).astype(np.float)/(np.sum(gt_edge).astype(np.float) + 1e-8)
recall_edge

# f1 = (1 + \{beta}2) * precision * recall / \{beta}2 * precision + recall
f1_edge = (1 + 0.3) * precision_edge * recall_edge / (0.3 * precision_edge + recall_edge + 1e-8)
f1_edge
xuebinqin commented 3 years ago

Please try to output the results of skimage.morphology.erosion and visually double check that. You will find the reason.

On Dec 29, 2020, at 8:50 PM, Cecilia notifications@github.com wrote:

Hello, I used the code you offered to calculate the f_relax_measure between my gt and pred mask, I doubt the way that why the disk size is set to 1, cause under that occasion, the erosion may always be zero?? Is there something I misunderstood???

def boundary_extraction(mask, k_size=1): """ 对结果进行腐蚀，进而得到边缘 """

(h, w) = mask.shape
mask_pad = np.zeros((h+10, w+10))
mask_pad[5:h+5, 5:w+5] = mask

mask_pad_erd = ndimage.grey_erosion(mask_pad, size=(k_size, k_size))
mask_pad_edge = np.logical_xor(mask_pad, mask_pad_erd)

return mask_pad_edge[5:h+5, 5:w+5]

pred = cv2.imread( '/mnt/work/yuyanpeng/code/personal/U-2-Net/test_data/u2net_results/ILSVRC2012_test_00052503.png', 0)

gt = cv2.imread( '/mnt/work/yuyanpeng/code/personal/PoolNet/data/DUTS-TE/DUTS-TE-Mask/ILSVRC2012_test_00052503.png', 0)

gt[gt <= 2np.mean(gt)] = 0 gt[gt > 2np.mean(gt)] = 1

pred[pred <= 127] = 0 pred[pred > 127] = 1

pred_edge = boundary_extraction(pred) gt_edge = boundary_extraction(gt)

distance_transform_edt 用于距离转换，计算图像中非零点到最近背景点（即0）的距离

这里先将背景和前景互换，即背景变成1，前景变成0，则变为计算背景中距离最近边缘的距离

pred_dist = distance_transform_edt(np.logical_not(pred_edge)) gt_dist = distance_transform_edt(np.logical_not(gt_edge))

The relaxed boundary recall (relaxRecallb) measures the fraction of ground truth boundary pixels

that are within ρ pixels of predicted boundary pixels. Here ρ sets to 3.

buffer = 3 pred_edge_buffer = np.zeros_like(pred_edge) pred_edge_buffer = pred_edge.copy()

precision = TP / 总数

pred_edge_buffer[gt_dist > buffer] = 0 # buffer 之外的点（即距离最近的边缘超过3个像素的点不考虑在内） precision_edge = np.sum(pred_edge_buffer).astype(np.float)/(np.sum(pred_edge).astype(np.float) + 1e-8) precision_edge

recall = TP / (TP + FN)

gt_edge_buffer = np.zeros_like(gt_edge) gt_edge_buffer = gt_edge.copy() gt_edge_buffer[pred_dist > buffer] = 0 # recall_edge = np.sum(gt_edge_buffer).astype(np.float)/(np.sum(gt_edge).astype(np.float) + 1e-8) recall_edge

f1 = (1 + {beta}2) precision recall / {beta}2 * precision + recall

f1_edge = (1 + 0.3) precision_edge recall_edge / (0.3 * precision_edge + recall_edge + 1e-8) f1_edge — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NathanUA/BASNet/issues/20#issuecomment-752318185, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSGORPCGKTXVYILG3TABADSXKPR3ANCNFSM4JAEBORQ.

vionwinnie commented 2 years ago

I have incorporated the relax-F measure in this forked version of toolkit for salient object detection: https://github.com/vionwinnie/SOD_Evaluation_Metrics

Thank you so much for providing the source code. I learnt a lot just by browsing through the issues and the responses.

xuebinqin commented 2 years ago

U're very welcome. That is a nice repo. Good job.

On Wed, May 4, 2022 at 12:52 PM Winnie Yeung @.***> wrote:

I have incorporated the relax-F measure in this forked version of toolkit for salient object detection: https://github.com/vionwinnie/SOD_Evaluation_Metrics

Thank you so much for providing the source code. I learnt a lot just by browsing through the issues and the responses.

— Reply to this email directly, view it on GitHub https://github.com/xuebinqin/BASNet/issues/20#issuecomment-1117689734, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSGORJQFWCMJBAWV2XBHTLVILBN5ANCNFSM4JAEBORQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Xuebin Qin PhD Department of Computing Science University of Alberta, Edmonton, AB, Canada Homepage: https://xuebinqin.github.io/