利用CAMELYON16提供的Annotations在測試時無法正確計算FROC

[christinesu12]

每個xml所產生的mask.tif經過computeEvaluationMask皆只有返回一個腫瘤 ex . test_004.xml具有3個腫瘤註釋，但是生成的mask tif文件返回1個腫瘤 xml to tif reader = mir.MultiResolutionImageReader() mr_image = reader.open('../images/test_021.tif') annotation_list = mir.AnnotationList() xml_repository = mir.XmlRepository(annotation_list) xml_repository.setSource('test_021.xml') xml_repository.load() annotation_mask = mir.AnnotationToMask() camelyon17_type_mask = False label_map = {'metastases': 1, 'normal': 2} if camelyon17_type_mask else {'_0': 1, '_1': 1, '_2': 0} conversion_order = ['metastases', 'normal'] if camelyon17_type_mask else ['_0', '_1', '_2'] output_path = "test_021_mask.tif" annotation_mask.convert(annotation_list, output_path, mr_image.getDimensions(), mr_image.getSpacing(), label_map, conversion_order)

evaluation mask def computeEvaluationMask(maskDIR, resolution, level): """Computes the evaluation mask.

Args:
    maskDIR:    the directory of the ground truth mask
    resolution: Pixel resolution of the image at level 0
    level:      The level at which the evaluation mask is made

Returns:
    evaluation_mask
"""
slide = openslide.open_slide(maskDIR)
dims = slide.level_dimensions[level]
pixelarray = np.zeros(dims[0]*dims[1], dtype='uint')
pixelarray = np.array(slide.read_region((0,0), level, dims))
distance = nd.distance_transform_edt(255 - pixelarray[:,:,0])
Threshold = 75/(resolution * pow(2, level) * 2) # 75µm is the equivalent size of 5 tumor cells
binary = distance < Threshold
filled_image = nd.morphology.binary_fill_holes(binary)
evaluation_mask = measure.label(filled_image, connectivity = 2) 
return evaluation_mask

[yil8]

@christinesu12 你好，用的不是基于xml来生产mask.tif的，是用的以前官方提供的test.tif，但是现在似乎被删除了。

[christinesu12]

您好，請問有那邊可以下載之前官方提供的tif 或者能夠提供你們測試的test set或是mask嗎？ 希望可以驗證是否能夠以你們的方式做出一樣的結果，非常感謝您的回覆

[yil8]

@christinesu12 可以留一下你的email吗？我之前保留两个test_026.tif可以email给你测试一下

[lvguofeng1303]

你好 我使用ASAP查看了一下XML文件 有的标注group是以tumor为组名 而不仅仅是“_0”,"_1"和“_2”，但是似乎正确生成的mask.tif 也无法正确计算FROC。 那个请问哪可以下载官方提供的tif吗，我也想测试一下

[yil8]

@lvguofeng1303 你好，我之前用的是官方提供的test.tif，不是用ASAP转XML生成的，但是后来似乎官方把这个test.tif给删除了。

[lvguofeng1303]

@yil8 你好，我通过ASAP查看了一下全部test的XML文件，似乎使用这一组代码可以正确的生成mask.tif： `def process(file): reader = mir.MultiResolutionImageReader()

file_name = os.path.splitext(file)[0]
path1 = os.path.join("/staff/lvguofeng/Camelyon16_dataset/testing/images", file_name+".tif")
path2 = os.path.join("/staff/lvguofeng/Camelyon16_dataset/testing/test_xml", file_name+".xml")

mr_image = reader.open(path1)
annotation_list = mir.AnnotationList()
xml_repository = mir.XmlRepository(annotation_list)
xml_repository.setSource(path2)
xml_repository.load()
annotation_mask=mir.AnnotationToMask()
camelyon17_type_mask = False
label_map = {'metastases': 1, 'normal': 2} if camelyon17_type_mask else {'_0': 255, '_1': 255, '_2': 0, 'Tumor': 255, 'Exclusion': 0}
conversion_order = ['metastases', 'normal'] if camelyon17_type_mask else  ['_0', '_1', '_2', 'Tumor', 'Exclusion']
output_path= os.path.join("./tumor_mask/test/", file_name+"_mask.tif")
annotation_mask.convert(annotation_list, output_path, mr_image.getDimensions(), 
                        mr_image.getSpacing(), label_map, conversion_order)

global lock
global count

with lock:
    count.value += 1
    print("-------------- Done {}, name is {}".format(count, file_name))`

但是我依然没办法得到如论文所说的FROC值，直接跑Evaluation_FROC.py得到的结果是： Avg FP = 0.25 Sensitivity = 0.1963470319634703 Avg FP = 0.5 Sensitivity = 0.2207001522070015 Avg FP = 1 Sensitivity = 0.2968036529680365 Avg FP = 2 Sensitivity = 0.3926940639269406 Avg FP = 4 Sensitivity = 0.547945205479452 Avg FP = 8 Sensitivity = 0.7442922374429224 Avg Sensivity = 0.3997970573313039 我想问问你们是否在获取坐标的过程，也就是nms.py这份代码，还使用了什么后处理方案？ 十分感谢您的回答 ps. 这个是我利用你的代码跑出来的test_026的结果，应该训练好的模型和你的结果没多大区别

[yil8]

@lvguofeng1303 我没有做nms之外的后处理，所有的流程都记录在README里面了哈

[Doloresru]

您好，请问您后来这个问题解决了吗？

[XiaoXueShengwangrui]

请问如何获取test.tif的GT？是否能使用官方的test数据集提供的.xml来生产测试集的GT图呢？

[lzx325]

It is possible to use official XML files to generate masks and then evaluate. My code for the conversion is:

reader = mir.MultiResolutionImageReader()
mr_image = reader.open(tiff_file)
assert mr_image is not None
annotation_list = mir.AnnotationList()
xml_repository = mir.XmlRepository(annotation_list)
xml_repository.setSource(xml_file)
xml_repository.load()
annotation_mask = mir.AnnotationToMask()
camelyon17_type_mask = False
monitor=mir.CmdLineProgressMonitor()
annotation_mask.setProgressMonitor(monitor)
label_map = {'metastases': 255, 'normal': 0} if camelyon17_type_mask else {'_0': 255, '_1': 255, 'Tumor':255, '_2': 0, 'None':0,'Exclusion':0}
conversion_order = ['metastases', 'normal'] if camelyon17_type_mask else  ['_0', '_1', 'Tumor','_2','None','Exclusion']
annotation_mask.convert(annotation_list, output_path, mr_image.getDimensions(), mr_image.getSpacing(), label_map, conversion_order)

Please note that

1. Some annotations are under the group 'Exclusion' and they should be regarded as non-tumor. It seems that under the group '_2' should also be considered as non-tumor.

2. I have converted the tumor region pixel value to 255 since it is expected in the evaluation script Evaluation_FROC.py.

   def computeEvaluationMask(maskDIR, resolution, level):
   """Computes the evaluation mask.
   
   Args:
       maskDIR:    the directory of the ground truth mask
       resolution: Pixel resolution of the image at level 0
       level:      The level at which the evaluation mask is made
   
   Returns:
       evaluation_mask
   """
   distance = nd.distance_transform_edt(255 - pixelarray[:,:,0]) # lizx: reverse those masks, note the 255 assumption!!
   Threshold = 75/(resolution * pow(2, level) * 2) # 75µm is the equivalent size of 5 tumor cells # lizx: keep the margin within 5 tumor cells positive
   binary = distance < Threshold
   filled_image = nd.morphology.binary_fill_holes(binary) # lizx: this is the mask for this level
   evaluation_mask = measure.label(filled_image, connectivity = 2) # lizx: get connected components
   return evaluation_mask

3. The annotations of some slides are difficult to convert and take a lot of time, e.g. test_021. Please be patient and wait for it to finish. Otherwise, all the model's predictions on this slide will be regarded as false positives. And this will seriously affect the score.
4. The annotation files do not contain the annotation for test_114. You should exclude it when evaluating the model. Otherwise, all of the model's correct predictions will be incorrectly treated as false positives. My final result looks like this:

   Avg FP =  0.25
   Sensitivity =  0.6814159292035398
   Avg FP =  0.5
   Sensitivity =  0.7433628318584071
   Avg FP =  1
   Sensitivity =  0.7920353982300885
   Avg FP =  2
   Sensitivity =  0.8407079646017699
   Avg FP =  4
   Sensitivity =  0.8761061946902655
   Avg FP =  8
   Sensitivity =  0.9026548672566371
   Avg Sensivity =  0.806047197640118

[yil8]

@lzx325 WOW, that's pretty cool. Do you mind make a pull request to add this part to README?