Training UNETR model on ACDC dataset

[programmerZe]

Hi, thanks for your great work. I've tried to train UNETR model on ACDC dataset using the pretrained checkpoint. But I got error: "ValueError: The size of the proposed random crop ROI is larger than the image size." Could you please give me a solution? My training command is: python main.py --batch_size=1 --logdir=unetr_pretrained --optim_lr=1e-4 --lrschedule=warmup_cosine --infer_overlap=0.5 --save_checkpoint --data_dir=../data/ACDC_training/ --pretrained_dir='./pretrained_models/' --pretrained_model_name='UNETR_model_best_acc.pth' --resume_ckpt --out_channels 4 --roi_x 256 --roi_y 256 --roi_z 16

Full error message:

Traceback (most recent call last):
  File "main.py", line 245, in <module>
    main()
  File "main.py", line 106, in main
    main_worker(gpu=0, args=args)
  File "main.py", line 239, in main_worker
    post_pred=post_pred,
  File "/home/zy/UNTER/trainer.py", line 179, in run_training
    model, train_loader, optimizer, scaler=scaler, epoch=epoch, loss_func=loss_func, args=args
  File "/home/zy/UNTER/trainer.py", line 57, in train_epoch
    for idx, batch_data in enumerate(loader):
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/transform.py", line 92, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/transform.py", line 60, in _apply_transform
    return transform(parameters)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/croppad/dictionary.py", line 1136, in __call__
    self.randomize(label, fg_indices, bg_indices, image)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/croppad/dictionary.py", line 1126, in randomize
    self.spatial_size, self.num_samples, self.pos_ratio, label.shape[1:], fg_indices_, bg_indices_, self.R
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/utils.py", line 482, in generate_pos_neg_label_crop_centers
    centers.append(correct_crop_centers(center_ori, spatial_size, label_spatial_shape))
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/utils.py", line 406, in correct_crop_centers
    raise ValueError("The size of the proposed random crop ROI is larger than the image size.")
ValueError: The size of the proposed random crop ROI is larger than the image size.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/transform.py", line 92, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/transform.py", line 60, in _apply_transform
    return transform(parameters)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/compose.py", line 160, in __call__
    input_ = apply_transform(_transform, input_, self.map_items, self.unpack_items)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/transform.py", line 116, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.croppad.dictionary.RandCropByPosNegLabeld object at 0x7f7632c09e48>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/data/dataset.py", line 95, in __getitem__
    return self._transform(index)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/data/dataset.py", line 652, in _transform
    return super()._transform(index)
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/data/dataset.py", line 81, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/home/zy/miniconda3/envs/unter/lib/python3.6/site-packages/monai/transforms/transform.py", line 116, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f7632c09828>

[tangy5]

The Swin UNETR model uses 96x96x96 patch by default in the setting. I guess the data original volume is smaller than this size. You can add a "SpatialPadd" transform before the "RandCropByPosNegLabeld" transform here:

https://github.com/Project-MONAI/research-contributions/blob/c3c935b9ea7544019dd5e4141ac22a174e545ef7/SwinUNETR/BTCV/utils/data_utils.py#L84

SpatialPadd: https://docs.monai.io/en/stable/transforms.html#spatialpadd

[programmerZe]

It works. Thank you! And I got another problem: the type of my label is scribble, so there are some unlabelled pixels. I need to use the parameter ignore_index in loss function, but the DiceCELoss function does not support it and return error. How can I fix it? Thanks!

[tangy5]

Hi @programmerZe , I guess we have discussed this. I will close for now, feel free to reopen this if there are further problem.