The SegmentationReview extension is a powerful tool for clinicians who need to quickly and efficiently review deep-learning generated segmentations. With this extension, you can use a Likert-type rating system that scores segmentations on a four-point scale, ranging from "Acceptable with no changes" to "Unacceptable, not visible/wrong location".
One of the key benefits of SegmentationReview is the ability to load the entire dataset with all masks and volumes in one batch. This simplifies the workflow and eliminates the need to load masks and volumes separately. Additionally, the results of the review are saved as a csv file, which can be easily used for further analysis.
NOTE: If you have not finished the review of all the masks in one go, feel free to close and resume the extension later on. The last annotated file would be restored from the annotations.csv file saved in the path. (That being said, this won't work if you annotated cases and then added new ones to the same folder. Keep new cases in the separate folder once you already started annotations to avoid conflicts)
Overall, SegmentationReview provides a streamlined solution for clinicians who want to speed up the validation process of deep-learning generated segmentations. It is a valuable tool that can help to improve workflow efficiency and reduce the burden associated with segmentation validation.
Note: available for 3D Slicer 5.3.0 marketplace and later. Updated version with no mask support, mapper.csv and centering of masks is available in 3D Slicer 5.5.0 (currently preview version) and later.
If you found it useful in your work please cite the paper: Anna Zapaishchykova, Divyanshu Tak, Aidan Boyd, Zezhong Ye, Hugo J.W.L. Aerts, Benjamin H. Kann "SegmentationReview: A Slicer3D extension for fast review of AI-generated segmentations" https://doi.org/10.1016/j.simpa.2023.100536
You can install SegmentationReview
from the 3D Slicer extensions manager from version 5.3.0.
To open the extension manager, go to View -> Extensions Manager
in the main menu. In the extensions manager, search for SegmentationReview
and click Install
to install the extension. Once the extension is installed, switch to the SegmentationReview
module by clicking Modules -> SegmentationReview
in the main menu.
Panel Overview:
The interface displays three main panels:
Results:
AI-generated segmentation ratings are automatically saved as a .csv file, which can be further analyzed using any tool of choice. The example above presents a dummy dataset using MS Office, but any other analysis tool can be used.
The annotation.csv
has a following structure (with no header):
Filename | Rating | Comments | Mask_filename | Mask_status |
---|---|---|---|---|
image1.nii.gz | Acceptable with no changes | image1_mask.nii.gz | Mask loaded | |
image2.nii.gz | Bad images | wrong sequence | No mask | |
... | ... | ... |
"Rating" (2nd column) has the following encoding: (1) Acceptable with no changes, (2) Acceptable with minor changes, (3) Unacceptable with major changes, (4) Unacceptable and not visible, and (5) Bad images.
"Mask_status" (5th column) has the following encoding: (2) Mask loaded, (1) Cannot load mask and (0) No mask
To get started, create a folder with the following file structure:
- image1.nii.gz
- image1_mask.nii.gz
- image2.nii.gz
- image2_mask.nii.gz
- image3.nii.gz
...
The images should be in NIfTI format (.nii.gz
), with corresponding segmentation masks labeled _mask.nii.gz
. If masks are not available, you can still use the extension (images would be loaded for review without the mask), but you will not be able to edit the masks.
An example dataset of T1w brain scans and their corresponding brain segmentations is provided in the example_dataset_no_mapping
folder with already reviewed images. You can use this dataset to test the extension.
Load the dataset into 3D Slicer: After starting 3D Slicer, open File -> Add Data
from the main menu, then select the folder containing the images and masks and press OK
. After loading, you will see how many images are loaded under the Checked status. If the path is opened for the first time, an annotations.csv
file will be created in the same folder. This file will contain the results of the rating and will be automatically updated after each rating. Additionally, the annotations.csv
file allows you to restore the annotation process in case of a crash or if there are too many images to rate in one session.
Assign a Likert score to each image: In the "SegmentationReview" module, click on the Likert rating
tab, select the image, and then select a rating from the drop-down menu (ranging from Acceptable with no changes to Unacceptable and not visible). When you're done, click Save and Next
to move to the next image. The results will be automatically saved in the annotations.csv
file.
Optional: Edit the mask of an image: If you want to change the segmentation mask of any image, you can use the "Segment Editor" module that is added to the extension. Select the Segment editor
tab, edit the mask using the brushes or eraser, and then click Overwrite edited mask
to save the new mask. The new mask will be saved in the same folder as the original mask, with timestamp added to the end of the name of mask.
This is a more flexible option that allows to map the filenames in the folder to the filenames in the mapping file. This is useful when the filenames in the folder are not the same as in the mapping file.
Create a mapping.csv with the following structure (with header):
img_path,mask_path
sub-pixar008_T1w/sub-pixar008_T1w.nii.gz,sub-pixar008_T1w/sub-pixar008_T1w_mask.nii.gz
sub-pixar009_T1w.nii.gz,sub-pixar009_T1w_mask.nii.gz
sub-pixar0010_T1w.nii.gz,
...
See an example in the example_dataset_with_mapping
folder. Note: this would also work if the mask_path column is empty. In this case, the masks would not be loaded, but the images would still be loaded for review. Then, follow the steps 2-4 from the previous section "OPTION 1: NO mapping file".
This is useful when you have multiple masks per subject and you want to find the "first" best one(Likert score 1). Create a mapping_unique.csv:
subj_id, img_path, mask_path
id1_day1, sub-pixar009_T1w.nii.gz,
id1_day1, sub-pixar010_T1w.nii.gz,
...
See an example in the example_dataset_with_mapping_unique
folder.
This extension is available from the extensions marketplace starting v.5.3.0. Here are the steps to install the extension from the source and develop the extension locally. This is useful for testing and contributing changes leveraging the GitHub pull request contribution workflow.
Download the source code using git
/ OR download the .zip and unpack it
git clone git@github.com:zapaishchykova/SegmentationReview.git
(Optional) Enable the developer mode will allow to reload the module from source without having to restart the application.
The extension will pop up under the Modules:"Examples" section.
The example data was obtained from the OpenfMRI databaset. Its accession number is ds000228. Brains were segmented using HD Brain Extraction tool.
Version available in 3D Slicer 5.3.0-5.5.0 marketplace does not include "no masks" support. Updated version with no mask support, mapper.csv and centering of masks is available currently in the preview release of 3D Slicer 5.5.0.
We would like to thank these authors for their significant contributions to the project:
This extension is distributed under the terms of the MIT license.