search

paip-2019 / challenge

30 stars 10 forks source link

Assessment Metrics - What is HT(Jaccard Index; 0.65)? #24

Closed yiping-wang closed 5 years ago

yiping-wang commented 5 years ago

Thanks for providing and supporting this challenge!

Task 1: Cancer Segmentation Evaluation Metrics: Jaccard Index

  • Jaccard Index * HT(Jaccard Index; 0.65) where HT(Jaccard Index; 0.65) = 1
  • f Jaccard Index >= 0.65 and HT(Jaccard Index; 0.65) = 0 otherwise.
  • Missing data will be scored as 0

Would you please explain what is HT(Jaccard Index; 0.65) in more details? To my best knowledge, the only abbreviation for HT I can think of is Hough Transform.

Also, to make sure I understand the Evaluation Metrics correctly, we should compute the Jaccard index for the prediction mask and the ground truth mask, right? For example, one WSI image has a 45561 43823 TIF ground truth mask, we should compute the Jaccard Index of prediction, which is also 45561 43823 mask with the ground truth?

hjoonjang commented 5 years ago

@yiping-wang Thank you for your patience. HT is a kind of high-pass function which passes through the result when it is bigger than 0.65 and ignores (makes it 0.0) the result otherwise. (It may represent Head/Tail.) The reason why we set HT is that we'd like to have the scores of participants vary more on the leaderboard because of the lower scores.

Of the second part, I think you are confused now. Jaccard Index is a kind of evaluation metrics by itself, which we utilize for measuring binary_img-to-binary_img similarity score, so you don't have to calculate Jaccard Index for yourself. When you submit a 45561 * 43823 prediction binary mask image, for example, then the leaderboard would automatically calculate your score. The given description is just for giving detailed information about the implementation of the evaluation metric. In short, we use the Jaccard Index as image-wise evaluation, not as pixel-wise one.

Two additional information:

  1. The HT threshold value of 0.65 might be subject to change somehow later.
  2. We are actually considering to amend the descriptions because it could have been clearer. Thanks for your feedback.