Shifted patches when merging patch predictions!

[mrahimpour]

• TIA Toolbox version: 1.3.2
• Python version: 3.9
• Operating System: Ubuntu 22.04.1 LTS

Description

I am using the tiatoolbox to implement the automatic tumor contouring pipeline for lung H&E images. The pipeline includes the following steps:

• Patch extraction
• Patch prediction
• Prediction merging

When merging the predictions, in some cases, the patches are not located at the right coordinate and they are clearly shifted. Also, in very few cases, coordinated_list and predicted_patches have different sizes. I was wondering if you faced this issue before or have any idea why this issue is happening.

Here is the code I use for prediction merging:

#read WSI and calculate the wsi_size at the working resolution
wsi_reader= WSIReader.open(input_img=wsi_path)
wsi_size= wsi_reader.slide_dimensions(resolution=working_resolution,units="mpp")

#read the mask and calculate the scale factor
mask_img= tifffile.imread(mask_path)
scale_factor= np.array(wsi_size) / mask_img.shape[:2][::-1]

#create the patchextractor object with the working_resolution used to train the model
fixed_patch_extractor= patchextraction.get_patch_extractor(input_img=wsi_reader,
method_name="slidingwindow",
 patch_size=patch_size,
 input_mask=mask_img,
 resolution=working_resolution,
 stride=stride,
 units="mpp")

#get the coordinate list for the patches
coordinate_list= fixed_patch_extractor.get_coordinates(
 image_shape=wsi_size,
 patch_input_shape=(4096,4096),
 stride_shape=(4000,4000),
)
#generate the virtual wsi reader for the mask image at the working resolution,
#the mask reader will be used to correct the the coordinate_list
mask_data_resized= imresize(mask_img,scale_factor)
mask_reader= VirtualWSIReader(mask_data_resized,mpp=wsi_reader.info.mpp[0])

#filter the coordinates if a mask is used for patch extraction
if mask_img is not None:
 selected_coord_indices =fixed_patch_extractor.filter_coordinates_fast(
 mask_reader,
 coordinate_list,
 coordinate_resolution=wsi_reader.info.mpp[0],
 coordinate_units="mpp")
 corrected_coordinate_list =coordinate_list[selected_coord_indices]

#read the patch predictions
predictions= []
for pred in sorted(os.listdir(prediction_path)):
 prediction = imread(pre)[:,:,0]
 predictions.append(prediction)

#create the patch predictor object to call merge_predictionfunction
output= {
 'resolution':working_resolution,
 'units':'mpp',
 'predictions':np.asarray(predictions),
 'coordinates':np.asarray(corrected_coordinate_list),
}

#when generating the patches, we apply padding.
#Therefore, we need to consider it when merging the predictions
#later we will get back to the original wsi_size before downsampling the prediction
wsi_size_padded= (coordinate_list[-1][2],coordinate_list[-1][3]) 
merged_img= np.transpose(np.zeros(wsi_size_padded))
merged_reader= VirtualWSIReader(merged_img,mpp=working_resolution)
merged= PatchPredictor.merge_predictions(merged_reader,
 output,
 resolution=working_resolution,
 units='mpp')

save_path= os.path.join(output_folder,sub+ "_merged_ds.tif")

#get back to the wsi_size by removing the padded part
modified_merged= merged[0:wsi_size[1],0:wsi_size[0]]

#down-sample the merged prediction to be the same size as the GT mask
modified_merged_ds= imresize(modified_merged,1/scale_factor)
io.imsave(save_path,modified_merged_ds,check_contrast=False)

[shaneahmed]

We are working on a new design for engines in #578. For now you can look at scale_factor while converting the annotations https://github.com/TissueImageAnalytics/tiatoolbox/blob/336d5e53a3f597125a442e5d6a3148944eca1e60/tiatoolbox/utils/misc.py#L894

[shaneahmed]

Partially fixed in #777