search

M3DV / RibFrac-Challenge

MICCAI 2020 RibFrac Challenge: Rib Fracture Detection and Classification (3D Instance Segmentation)
https://ribfrac.grand-challenge.org/
Apache License 2.0
49 stars 10 forks source link

About the annotation of the datasets #6

Closed SwanKnightZJP closed 4 years ago

SwanKnightZJP commented 4 years ago

Hi, dear organizer.

During verifying the performance of our method, we meet some problems with annotations in Training set and Validation Set. It mainly includes the following two aspects:

  1. It seems that some fracture areas in the Validation Set are not annotated. As shown in the figure, there are two abnormal cortical bones, but the annotated area does not include these two places. image image
  2. Part of the fracture occurred on multiple ribs, but only have one label. Such as the RibFrac114 in Training set and RibFrac445 in Validation Set. This may be for the convenience of annotating. However, annotations like this with gaps in the middle may have problems while under evaluation. (see line 246 in evaluation.py for details) When the centroid(x, y, z) is located in these gaps, whether it exists possible for the original labels to be classified as the 'Background'? image image
duducheng commented 4 years ago

Hi @SwanKnightZJP ,

Thank you very much for your feedback!

For both issue 1 and 2, I am sorry that the mentioned issues are possible, even though we have tried our best to improve the annotation quality. It may occurs in the human annotation procedure or the pre-processing step.

We have paid more attention on the test set (than the training and validation), so that the risk of these issues is largely reduced. It is still human annotations anyway, I can not guarentee the 100% accuracy of the annotations though.

We are re-checking the annotations with the hope of a updated version (RibFrac v2), but it is not possible to make it done for the RibFrac Challenge this year. Stay tuned!

Please note that it is also fair the participants this year, since every participants use the same dataset anyhow. Thanks for understanding.

Jiancheng Yang On behalf of MICCAI 2020 RibFrac Challenge Organizers

SwanKnightZJP commented 4 years ago

Thank you very much for your reply, and thanks to the RibFrac project team for publishing such an excellent dataset. Looking forward to the updated version. (╹◡╹)ノ♡