Open sander-boelders opened 2 years ago
tangy5
commented
2 years ago Hi @sander-boelders , thanks for trying SwinUNETR. This error logs are related to the Crop transformation. Could you double check whether the pre-processed data are under same configuration as the 2021 data with SwinUNETR? I notice the data folder might be pre-processed by nnUNet pipeline, which may be different from MONAI transforms. Thank you.
sander-boelders
commented
2 years ago Hi @tangy5, thank you for the quick response!
The datafolder contains the raw T1c images from BraTS2020 at a 240x240x155 resolution, equivalent to the resultion described in the paper. I downloaded the BraTS2021 data and ran into the same problem.
The problem seems to be using only 1 modality. When I use 2 modalities it works as expected. I formatted my .json as a single image "image":"BraTS2021_01016/BraTS2021_01016_t1ce.nii.gz" and as a list with one "image":["BraTS2021_01016/BraTS2021_01016_t1ce.nii.gz"]. Both resulted in the exact same error.
tangy5
commented
2 years ago Hi @sander-boelders thanks for the follow-ups. I'm happy to dig into this more. First, I saw the command used pre-trained weights, is the pre-trained checkpoint the one published with 4 channel input? If the problem still exists, could you construct a sample code for reproducing the error (minimum code block for reproducing same error), so that I can tested the dataloader and transform on my local side? Thank you.
sander-boelders
commented
2 years ago Hi @tangy5, I dont think I am using pre-trained weights. --save_checkpoint is documented as saving checkpoints during training --use_checkpoint is documented as use gradient checkpointing to save memory I am not using --checkpoint CHECKPOINT documented as start training from saved checkoint.
Reducing my training command to a minimum did not do the trick: python main.py --feature_size=48 --batch_size=1 --json_list=/PATH_TO_BRATS21/files_t1c.json --data_dir=/Data/users/vivian/PATH_TO_BRATS21/ --in_channels=1 --out_channels=4 --spatial_dims=3 --roi_x=128 --roi_y=128 --roi_z=128
To reproduce this you can use the files_t1c.json (https://drive.google.com/file/d/1UYbvTe_fviBhwlNapYB2_XPcQykIWoEt/view?usp=sharing) file which points to just the T1c files and place it in the folder with the BraTS2021 data. After this, you can run the command as written above with PATH_TO_BRATS21 replaced with the path to the BraTS21 dataset.
When you set --in_channels=2 and add a second modality to the .json file it runs without problems.
Thanks for your help.
Kind regards, Sander
vsshu
commented
2 years ago Dear @tangy5 and @sander-boelders ,
I was wondering if some progress has been made as I am currently experiencing a similar issue with the crop transformation.
Kind regards, Vivian
python main.py --feature_size=48 --batch_size=1 --logdir=unetr_test_dir --fold=0 --optim_lr=1e-4 --lrschedule=warmup_cosine --infer_overlap=0.5 --save_checkpoint --val_every=10 --json_list=
Traceback (most recent call last):
File "/home/vivian/miniconda3/envs/swin/SwinUNETR/BRATS21/main.py", line 230, in
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/vivian/.local/lib/python3.9/site-packages/monai/transforms/transform.py", line 89, in apply_transform return _apply_transform(transform, data, unpack_items) File "/home/vivian/.local/lib/python3.9/site-packages/monai/transforms/transform.py", line 53, in _applytransform return transform(parameters) File "/home/vivian/.local/lib/python3.9/site-packages/monai/transforms/compose.py", line 173, in call input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats) File "/home/vivian/.local/lib/python3.9/site-packages/monai/transforms/transform.py", line 113, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.transforms.io.dictionary.LoadImaged object at 0x7ffa0e840f40>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/vivian/.local/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/home/vivian/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/vivian/.local/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
tangy5
commented
2 years ago @vsshu hi, based on the logs, there is an issue when loading the data, I guess there might be a memory issue which cause the process work failed.
Dear,
With great intrest I have read your paper on Swin UNETR for brain tumor segmentation as it seems to solve our exact problem when segmenting small tumors (using context vs having a high resolution)
For a research project, we are trying to segment tumors using T1c scans only on the BraTS 2020 data. Moreover, I am predicting only one region/category. I get the following error from the dataloader when the bounding box is computed: ValueError: Sequence must have length 2, got 3.
The BraTS2020 data has the same resolution as the 2021 data, so that can not cause the problem. Additinally, I believe to have set all the parameters accordingly (--in_channels=1 --out_channels=1 --spatial_dims=3 --roi_x=128 --roi_y=128 --roi_z=128) and the data loads fine.
Kind regards, Sander
Full traceback:
python main.py --feature_size=48 --batch_size=1 --logdir=unetr_test_dir --fold=0 --optim_lr=1e-4 --lrschedule=warmup_cosine --infer_overlap=0.5 --save_checkpoint --val_every=10 --json_list=/home/sander/pycharm_swinunetr/dataset.json --data_dir=/home/sander/Documents/nnUNet_data/nnUNet_raw/nnUNet_raw_data/Task603_BraTS2020/ --use_checkpoint --noamp --in_channels=1 --out_channels=1 --spatial_dims=3 --roi_x=128 --roi_y=128 --roi_z=128 0 gpu 0 Batch size is: 1 epochs 300 Total parameters count 62186659 Writing Tensorboard logs to ./runs/unetr_test_dir 0 Tue Aug 30 14:57:17 2022 Epoch: 0 Traceback (most recent call last): File "/home/sander/pycharm_swinunetr/main.py", line 234, in
main()
File "/home/sander/pycharm_swinunetr/main.py", line 100, in main
main_worker(gpu=0, args=args)
File "/home/sander/pycharm_swinunetr/main.py", line 218, in main_worker
accuracy = run_training(model=model,
File "/home/sander/pycharm_swinunetr/trainer.py", line 169, in run_training
train_loss = train_epoch(model,
File "/home/sander/pycharm_swinunetr/trainer.py", line 36, in train_epoch
for idx, batch_data in enumerate(loader):
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 681, in next
data = self._next_data()
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
return self._process_data(data)
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
data.reraise()
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/_utils.py", line 461, in reraise
raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/transform.py", line 89, in apply_transform
return _apply_transform(transform, data, unpack_items)
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/transform.py", line 53, in _apply_transform
return transform(parameters)
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/croppad/dictionary.py", line 889, in call
box_start, box_end = self.cropper.compute_bounding_box(img=d[self.source_key])
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/croppad/array.py", line 744, in compute_bounding_box
spatial_size = np.asarray(compute_divisible_spatial_size(orig_spatial_size.tolist(), k=self.k_divisible))
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/utils.py", line 1293, in compute_divisible_spatial_size
k = fall_back_tuple(k, (1,) * len(spatial_shape))
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/utils/misc.py", line 189, in fall_back_tuple
user = ensure_tuple_rep(user_provided, ndim)
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/utils/misc.py", line 146, in ensure_tuple_rep
raise ValueError(f"Sequence must have length {dim}, got {len(tup)}.")
ValueError: Sequence must have length 2, got 3.
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/transform.py", line 89, in apply_transform return _apply_transform(transform, data, unpack_items) File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/transform.py", line 53, in _applytransform return transform(parameters) File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/compose.py", line 173, in call input = apply_transform(transform, input, self.map_items, self.unpack_items, self.log_stats) File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/transform.py", line 113, in apply_transform raise RuntimeError(f"applying transform {transform}") from e RuntimeError: applying transform <monai.transforms.croppad.dictionary.CropForegroundd object at 0x7fd737c0ec50>
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop data = fetcher.fetch(index) File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 49, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/data/dataset.py", line 97, in getitem
return self._transform(index)
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/data/dataset.py", line 83, in _transform
return apply_transform(self.transform, data_i) if self.transform is not None else data_i
File "/home/sander/miniconda3/envs/swinunetr/lib/python3.10/site-packages/monai/transforms/transform.py", line 113, in apply_transform
raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7fd737c0e1a0>