This is the source code of the 1st place solution for segmentation task (with IoU 82.54%) in 2020 TN-SCUI challenge.
[Challenge leaderboardπ] [δΈζηreadmeπ]
Fold1 test results of DDTI dataset in colab
We use a simple cascaded framework for segmenting nodules, it can be easily extended to other single-target segmentation tasks.
Our pipeline is shown in the figure below.
In summary, what we do in our solution are:
step4_Merge.py
to merge 25 inference results into a final ensemble result by pixel-wised votingWe test our method on 2020 TN-SCUI training dataset(with 3644 images or nodules, malignant 2003 : benign 1641). The segmentation results of 5-fold CV based on "DeeplabV3+ with efficientnet-B6 encoder" are as following: | fold | Stage β | TTA at stage β | Stage β ‘ | TTA at stage β ‘ | DsC | IoU (%) |
---|---|---|---|---|---|---|---|
1 | β | 0.8699 | 79.00 | ||||
1 | β | β | 0.8775 | 80.01 | |||
1 | β | β | 0.8814 | 80.75 | |||
1 | β | β | β | 0.8841 | 81.05 | ||
1 | β | β | β | 0.8840 | 81.16 | ||
1 | β | β | β | β | 0.8864 | 81.44 | |
2 | β | β | β | β | 0.8900 | 81.99 | |
3 | β | β | β | β | 0.8827 | 81.07 | |
4 | β | β | β | β | 0.8803 | 80.56 | |
5 | β | β | β | β | 0.8917 | 82.07 |
We also test our method on another open access thyroid nodule ultrasound image dataset (DDTI public database [https://doi.org/10.1117/12.2073532], with 637 images or nodules after our preprocessing and data cleaning). The segmentation results of 5-fold CV based on "DeeplabV3+ with efficientnet-B6 encoder" are as following: | fold | Stage β | TTA at stage β | Stage β ‘ | TTA at stage β ‘ | DsC | IoU (%) |
---|---|---|---|---|---|---|---|
1 | β | 0.8391 | 74.80 | ||||
1 | β | β | 0.8435 | 75.49 | |||
1 | β | β | 0.8380 | 74.79 | |||
1 | β | β | β | 0.8406 | 75.16 | ||
1 | β | β | β | 0.8440 | 75.77 | ||
1 | β | β | β | β | 0.8469 | 76.17 | |
2 | β | β | β | β | 0.8392 | 74.57 | |
3 | β | β | β | β | 0.8756 | 79.61 | |
4 | β | β | β | β | 0.8131 | 72.48 | |
5 | β | β | β | β | 0.8576 | 77.74 |
You can follow the step-by-step tutorial below, or just download the pretrained weights and run
test_fold1.py
to reproduce our results. (note that we only provide the weights of 1st fold in 5-fold CV).
qxds
]qxds
]We have processed the DDTI dataset into the format of 2020 TN-SCUI datasetοΌyou can download it from
[GoogleDrive]
[BaiduWP, password:qxds
]
Fold1 test results of DDTI dataset in colab
We consulted the official organizer of 2020 TN-SCUI via email, and the official reply is that using of external dataset is not allowed in the 2020 TN-SCUI challenge, and participants are not allowed to provide the challenge data anywhere. Therefore, we hereby declare that we did not use any external datasets (such as the DDTI dataset) in the competition, and will not provide TNSCUI data anywhere. The DDTI dataset processed by us is only used as demo data for this code, so that everyone can run the entire scripts with a ready-made data.
This code can be easily performed on single-target segmentation tasks. Here, we split the whole process into 5 steps so that you can easily replicate our results or perform the whole pipeline on your private custom dataset.
step1_preprocessing.m
to perform the preprocessingstep2_TrainAndValidate.py
to train and validate the CNN modelstep3_TestOrInference.py
to testing model on original unprocessed imagesstep4_Merge.py
to get ensemble predicted resultYou should prepare your data in the format of 2020 TN-SCUI dataset, this is very simple, you only need to prepare:
Since the sponsor of TNSCUI challenge does not allow the dissemination of the 2020 TN-SCUI dataset, we give another thyroid ultrasound dataset demo for your reference. The demo dataset was derived from the DDTI open access dataset, and we have processed the dataset into the format of 2020 TN-SCUI datasetοΌyou can download it from
[GoogleDrive]
[BaiduWP, password:qxds
].
We have tested our code in following environmentοΌ
segmentation_models_pytorch
== 0.1.0 (with install command pip install segmentation-models-pytorch
)ttach
== 0.0.3torch
>οΌ1.0.0torchvision
imgaug
For installing segmentation_models_pytorch (smp)
, there are two commandsοΌ
pip install segmentation-models-pytorch
Use the above command, you can install the released version that can run with torch
<= 1.1.0, but without some decoders such as deeplabV3, deeplabV3+ and PAN.
pip install git+https://github.com/qubvel/segmentation_models.pytorch
Use the above command, you will install the latest version which includes decoders such as deeplabV3, deeplabV3+ and PAN, but requiring torch
>= 1.2.0.
We modified the original code of smp
, to make it can be run with torch
<= 1.1.0, and also have the latest version of the smp
library encoder such as deeplabV3, deeplabV3+ and PAN. The modified smp
is in the filefolder segmentation_models_pytorch_4TorchLessThan120
,
here are two ways to use it:
smp
with pip install segmentation-models-pytorch
firstly, then just copy the segmentation_models_pytorch_4TorchLessThan120
into your project folder to use itsegmentation_models_pytorch_4TorchLessThan120
into your project folder, then install these libs: torchvision>
=0.3.0, pretrainedmodels
==0.7.4, efficientnet-pytorch
>=0.5.1πThis step is optional, if you donot wanna preprocess image, just prepare your data as the demo format and skip to the Step2"training and validatation".
In step1, you should run the script step1_preprocessing.m
in MATLAB
to perform the preprocessing. For both TN-SCUI and DDTI dataset, there are some irrelevant regions to be removed. Because we used a cross-validation with a nodule size and category balance strategy, we should get the size of nodules for cross-validation.
After the code runs, you will get two folders and a csv file:
stage1
and stage2
, the data in folder stage1
is used to train the first network, which contains the preprocessed image with no irrelevant regions; and stage2
is used to train the second network, which contains the image in the expanded ROItrain.csv
with below dataformat, the size of a nodule is the number of pixels of the nodule after unifying preprocessed image to 256Γ256 pixelsid | cate | size |
---|---|---|
001.PNG | 0 | 2882 |
002.PNG | 1 | 3277 |
003.PNG | 1 | 3222 |
004.PNG | 1 | 256 |
β | β | β |
The example of preprocessed DDTI dataset was given here
[GoogleDrive]
[BaiduWP, password:qxds
]
In step2, you should run the script
step2_TrainAndValidate.py
to train the first or second network in the cascaded framework (you need to train the two networks separately), that is, the training process (such as hyperparameters) is the same for both first and second network except for the different training datasets used. With the csv file train.csv
, you can directly perform K-fold cross validation (default is 5-fold), and the script uses a fixed random seed to ensure that the K-fold cv of each experiment is repeatable (please note that the same random seed may cause different K-fold under different versions of scikit-learn
).
For example, you can train the first fold of the first network in two ways:
step2TrainAndValidate.py
and run the script.
""" just set your param as the default value """
# hyperparameters
parser.add_argument('--image_size', type=int, default=256) # recommended: 256 for 1st(stageβ
) network, 512 for 2nd(stage β
‘) networkπ
parser.add_argument('--batch_size', type=int, default=3)
parser.add_argument('--batch_size_test', type=int, default=30)
# custom parameters
parser.add_argument('--Task_name', type=str, default='dpv3plus_stage1_', help='DIR name,Task name')
parser.add_argument('--csv_file', type=str, default='./DDTI/2_preprocessed_data/train.csv')
parser.add_argument('--filepath_img', type=str, default='./DDTI/2_preprocessed_data/stage1/p_image')
parser.add_argument('--filepath_mask', type=str, default='./DDTI/2_preprocessed_data/stage1/p_mask')
parser.add_argument('--fold_K', type=int, default=5, help='number of cross-validation folds')
parser.add_argument('--fold_idx', type=int, default=1)
python step2_TrainAndValidate.py \
--batch_size=3 \
--Task_name='dpv3plus_stage1_'\
--csv_file='./DDTI/2_preprocessed_data/train.csv' \
--filepath_img='./DDTI/2_preprocessed_data/stage1/p_image' \
--filepath_mask='./DDTI/2_preprocessed_data/stage1/p_mask' \
--fold_idx=1
Something else in training process:
fold_idx
from 1 to K and running repeatedly, you can complete the training of all K folds. The results will be saved in the result
folder under the project. You can view the training log in real time through tensorboardX
filepath_img
and filepath_mask
) and Task_name
validate_flag
as True in the script step2TrainAndValidate.py
main()
with following code:
train_list = get_filelist_frompath(train_img_folder,'PNG')
train_list_GT = [train_mask_folder+sep+i.split(sep)[-1] for i in train_list]
test_list = get_filelist_frompath(test_img_folder,'PNG')
test_list_GT = [test_mask_folder+sep+i.split(sep)[-1] for i in test_list]
In fact, there is no need to re-run the script, because the test has been performed during the training process, and the detailed information of the result is recorded in the record.xlsx
or record.txt
. But you can also confirm the final training effect by running the script step2TrainAndValidate.py
after the training by setting the parameter mode
as test
.
It should be noted that the dataset validated in the training phase is preprocessed. Therefore, if you want to perform test based on the original unprocessed images, please refer to step3.
In step3, you should run the script
step3_TestOrInference.py
to perform inference based on the original unprocessed image.
By using the script step3_TestOrInference.py
, you can:
Here are some parameters should be noticed (Especially c1_size
and c2_size
):
fold_flag = False # False for inference on new data, and True for cross-validation
mask_path = None # None for inference on new data
save_path = None # If you don't want to save the inference result, set it to None
c1_size = 256 # The input image size for training the first network β
c1_tta = False # Whether to use TTA for the first network
orimg=False # False for no preprocessing to remove irrelevant areas
use_c2_flag = False # Whether to use the second network
c2_size = 512 # The input image size for training the second network β
c2_tta = False # Whether to use TTA for the second network
When you get multiple results after inference, you can use the script
step4_Merge.py
to integrate all the results. We use a pixel-wised voting method, and recommend voting on odd numbers of results to avoid fluctuations in making decision.
smg
and ttach
, all network and TTA used in this code come from his implement