image_id_list to run run.sh

[amsinha9]

In the Data Preparation step, what does the image_id_list correspond to? For example, in the 2018 training set under HGG, a folder is named Brats18_2013_2_1. So is the first entry in image_id_list "2013_2_1"? Also, do we retain the same folder structure that comes with the downloaded data (~\MICCAI_BraTS_2018_Data_Training\HGG..." or we get rid of the HGG and LGG sub-directories and combine all training data in a separate directory?

[jianpengz]

Just keep the original folder structure. The items in the image_id_list are the path to each MRI case, which are like

    BraTS2018/MICCAI_BraTS_2018_Data_Training/HGG/Brats18_2013_2_1
    BraTS2018/MICCAI_BraTS_2018_Data_Training/HGG/Brats18_2013_22_1
    ...

[amsinha9]

Okay, so within run.sh, data_dir is the path list BraTS2018/MICCAI_BraTS_2018_Data_Training/, correct? And for the below:

--train_list='list/train_list.txt' \ --val_list='list/val_list.txt' \

are train_list and val_list text files that include the subject ids for each respective group?

[jianpengz]

As said in the readme, the data and list should be put in './dataset'. In this case, data_dir should be '../dataset/'. And the list files do contain subject ids, for example, 'Brats18_2013_2_1' in the item 'BraTS2018/MICCAI_BraTS_2018_Data_Training/HGG/Brats18_2013_2_1' is the subject id.

[amsinha9]

Okay, to confirm, in the readme, it states "Put the data and image_id_list in dataset/", which means the original folder structure from BraTS should be moved to /dataset/. But we don't need an additional text file for image_id_list because the IDs will get parsed out.

It's just a little unclear to me because in run.sh, there is: --train_list='list/train_list.txt' \ --val_list='list/val_list.txt' \

but I can't find anywhere in the code that generates these specific text files, so I am not sure if we need to input those ourselves. So we don't need to supply formal train_list.txt and val_list.txt?

[jianpengz]

Yes, you need to generate the specific text files by yourself. Taking the BraTS for example, its data and list folder should be like

[amsinha9]

Okay great. Thank you again very much for clarifying all of this, much appreciated. One last question - within train_conresnet.py, do we need to edit lines 38-40 below:

parser.add_argument("--data_dir", type=str, default='/media/userdisk0/myproject-Seg/BraTS-pro/dataset/') parser.add_argument("--train_list", type=str, default='list/BraTS2018_old/train.txt') parser.add_argument("--val_list", type=str, default='list/BraTS2018_old/val.txt')

If I understand correctly, they should point to ~/dataset for the data and specific text files?

[jianpengz]

Yes. They pointe to your data and text files.

[amsinha9]

When I try to run run.sh, I get the error that snapshots/conresnet_log.file does not exist. In run.sh, do we need to change the following line:

--snapshot_dir='snapshots/conresnet/

It's not clear if we need to make these directories and keep them empty or if they need to point to relevant files. Can you clarify this as it was not addressed in the README?

[amsinha9]

Even if I comment out lines 7 and 18 in run.sh, this code doesn't run after changing run.sh and train_conresnet.py accordingly. What am I missing?

[jianpengz]

Do not comment out Line 7 and 18 in run.sh, which point to your path to save checkpoint and log file.

[amsinha9]

Okay, I believe I am missing something small. Within train_conresnet.py, I changed lines 38-40 by changing the default to point to my datasets and train_list.txt and val_list.txt Do I need to change the other lines within this script for the snapshots/conresnet (lines 41-42)? and how?

With keeping line 7 and 18 in run.sh, I get the error: snapshots/conresnet_log.file does not exist.

I believe I'm very close, so if you could clarify this, it should be good to go. Thanks!

[amsinha9]

Below is my run.sh script:

CUDA_VISIBLE_DEVICES=2,3 nohup python -m torch.distributed.launch --nproc_per_node=2 --master_port=$RANDOM train_conresnet.py \
--data_dir='~/ConResNet/dataset/' \
--train_list='list/train_list.txt' \
--val_list='list/val_list.txt' \
--snapshot_dir='snapshots/conresnet/' \
--input_size='80,160,160' \
--batch_size=2 \
--num_gpus=2 \
--num_steps=40000 \
--val_pred_every=2000 \
--learning_rate=1e-4 \
--num_classes=3 \
--num_workers=4 \
--random_mirror=True \
--random_scale=True \
> ~/ConResNet/dataset/logs/log.file 2>&1 &

Is there an error in my last line that is why I get the error " line 3: logs/log.file: No such file or directory" when I run run.sh?

[jianpengz]

It seems the folder 'logs/' does not exist. If so, try to create it first.

[amsinha9]

Here is my current folder structure:

|---dataset | ---| BraTS18 |------| MICCAI_BraTS_2018_Data_Training |------| MICCAI_BraTS_2018_Data_Validation | ---| list | ---| logs | ---| snapshots |------| conresnet

Does that all look correct?

[amsinha9]

Following up, isn't it supposed to output a log file at the end? Or is there a log.file I should have to run this?

[jianpengz]

Sorry for the late reply. The log file records the training log, like training loss, which would be created automatically. You can also comment this part out if unnecessary.