Getting training error unless I cache every training study

[AceMcAwesome77]

Hi, I am trying to get UNETR running on my own CT image dataset, but I am getting an error unless cache_num is greater than or equal to the number of training nifti files I have. The error is:

Training (X / X Steps) (loss=X.X): 0%| | 0/108 [00:00<?, ?it/s]Exception in thread Thread-27: Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 102, in apply_transform return _apply_transform(transform, data, unpack_items) File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 66, in _apply_transform return transform(parameters) File "/opt/conda/lib/python3.8/site-packages/monai/transforms/io/dictionary.py", line 154, in call data = self._loader(d[key], reader) File "/opt/conda/lib/python3.8/site-packages/monai/transforms/io/array.py", line 286, in call img = EnsureChannelFirst()(img) File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utility/array.py", line 231, in call raise ValueError(msg) ValueError: Metadata not available and channel_dim=None, EnsureChannelFirst is not in use.

My data structure is the same as the example MONAI test data, which seemed to work fine. If I add EnsureChannelFirst(strict_check=True, channel_dim=0),” to the train_transforms, then I get this error instead:

Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 102, in apply_transform return _apply_transform(transform, data, unpack_items) File "/opt/conda/lib/python3.8/site-packages/monai/transforms/transform.py", line 66, in _apply_transform return transform(parameters) File "/opt/conda/lib/python3.8/site-packages/monai/transforms/utility/array.py", line 235, in call img = MetaTensor(img) File "/opt/conda/lib/python3.8/site-packages/monai/data/meta_tensor.py", line 116, in new return torch.as_tensor(x, *args, **_kwargs).as_subclass(cls) # type: ignore RuntimeError: Could not infer dtype of dict

Adding AsChannelFirst to train_transforms does not work either. Does anyone know what effect would cause my training to work if every study is cached, but error otherwise?

Caching everything would not be a problem except my dataset is hundreds of volumes, and my GPU runs out of memory.

Thanks!

[KumoLiu]

Hi @AceMcAwesome77, could you please show what MONAI version you are using? From the error message seems you are not using MetaTensor which is supported after 0.9.1. Thanks!

[AceMcAwesome77]

Hi, thanks for the reply - I am using monai==1.1.0. So does that mean the version is fine?

[tangy5]

Can you paste the entire transform sequence? I guess some casting transform mess up the meta data. At the same time, you can check whether you need the "channel_dim=0", if your data is CT and do not contains channels, it can be "None" instead of 0. Thanks.

[AceMcAwesome77]

Here is my train_transforms function. You can see I've removed some of the random flipping and cropping from the example code to make it simpler, but it was giving the same error before that too.

train_transforms = Compose( [ LoadImaged(keys=["image", "label"], ensure_channel_first=True), ScaleIntensityRanged( keys=["image"], a_min=0, a_max=255, b_min=0.0, b_max=1.0, clip=True, ), Orientationd(keys=["image", "label"], axcodes="RAS"), Spacingd( keys=["image", "label"], pixdim=pixdim, mode=("bilinear", "nearest"), ), EnsureTyped(keys=["image", "label"], device=device, track_meta=False), RandCropByPosNegLabeld( keys=["image", "label"], label_key="label", spatial_size=patch_size, pos=1, neg=1, num_samples=num_samples, image_key="image", image_threshold=0, allow_smaller=True ) ] )

If I do LoadImaged(keys=["image", "label"], ensure_channel_first=None) instead, I get this error:

/opt/conda/lib/python3.8/site-packages/monai/transforms/spatial/array.py:704: UserWarning: axcodes ('RAS') length is smaller than the number of input spatial dimensions D=2. Orientation: input spatial shape is torch.Size([256, 80]), num. channels is 256,please make sure the input is in the channel-first format.

My input xy matrix size is 256x256 so it apparently wrongly thinks one of those is the channel... not sure why it only sees 2 dimensions. Should I add a dummy channel axis to the input 3D data? The example data is only 3 dimensions, so I tried to match that. But maybe it is cutting my 3D data to 2D for some reason, so I need to start with 4D?

[tangy5]

Thanks. Are your data in 2D format? Little confused here. If it's CT scans, in volume, it should be the same with the example tutorial. The tutorial adds the EnsureChannelFirstd after LoadImaged, Have you tried the same?

        LoadImaged(keys=["image", "label"]),
        EnsureChannelFirstd(keys=["image", "label"]),
        Orientationd(keys=["image", "label"], axcodes="RAS"),

[AceMcAwesome77]

It finally works! You pointed me out to two problems - the order of LoadImaged and EnsureChannelFirstd were swapped (I did not think the transform order mattered but it does), and I was using the function "EnsureChannelFirst" instead of EnsureChannelFirstd". The "Could not infer dtype of dict" error must have been because I didn't use the wrapper function that ends in "d".

If anyone comes across this error, here is an example of my working code:

train_transforms = Compose( [ LoadImaged(keys=["image", "label"]), EnsureChannelFirstd(keys=["image", "label"], strict_check=False, channel_dim='no_channel'), ScaleIntensityRanged( keys=["image"], a_min=0, a_max=255, b_min=0.0, b_max=1.0, clip=True, ), Orientationd(keys=["image", "label"], axcodes="RAS"), Spacingd( keys=["image", "label"], pixdim=(1.5, 1.5, 2.0), mode=("bilinear", "nearest"), ), RandCropByPosNegLabeld( keys=["image", "label"], label_key="label", spatial_size=(96, 96, 96), pos=1, neg=1, num_samples=4, image_key="image", image_threshold=0, allow_smaller=True ), ResizeWithPadOrCropd(keys=["image", "label"], spatial_size=(96, 96, 96), mode='constant') ] )