Exceptions while using my own dataset

[j-sieger]

I have my own nifti dataset for labelling. I could see the below Issue while starting the monaiLabel server.

[2022-02-21 05:04:44,855] [15675] [MainThread] [INFO] (monailabel.tasks.scoring.epistemic:110) - Using /home/ec2-user/efs/apps/deepedit/model/pretrained_dynunet.pt for running Epistemic
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 132, in <module>
    run_main()
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 123, in run_main
    result = a.scoring(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 307, in scoring
    return task(copy.deepcopy(request), datastore if datastore else self.datastore())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/tasks/scoring/epistemic.py", line 126, in __call__
    model, model_ts = self._load_model(self.model, self.network)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/tasks/scoring/epistemic.py", line 117, in _load_model
    model.load_state_dict(model_state_dict)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1482, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DynUNet:
        Unexpected key(s) in state_dict: "deep_supervision_heads.0.conv.conv.weight", "deep_supervision_heads.0.conv.conv.bias", "deep_supervision_heads.1.conv.conv.weight", "deep_supervision_heads.1.conv.conv.bias", "deep_supervision_heads.2.conv.conv.weight", "deep_supervision_heads.2.conv.conv.bias", "deep_supervision_heads.3.conv.conv.weight", "deep_supervision_heads.3.conv.conv.bias".

Facing Similar error after connecting from 3DSlicer and trying to fetch the next image. and also extra error ValueError: len(spatial_size) must be greater or equal to img spatial dimensions, got spatial_size=2 img=3.

[2022-02-21 05:18:00,001] [15313] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (ToTensord): Time: 0.0111; image: (1, 256, 256, 128)(float32); pred: torch.Size([1, 256, 256, 128])(torch.float32)
[2022-02-21 05:18:00,008] [15313] [MainThread] [ERROR] (uvicorn.error:380) - Exception in ASGI application
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 377, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/fastapi/applications.py", line 212, in __call__
    await super().__call__(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/endpoints/infer.py", line 177, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/endpoints/infer.py", line 160, in run_inference
    result = instance.infer(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 231, in infer
    result_file_name, result_json = task(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/tasks/infer.py", line 237, in __call__
    data = self.run_invert_transforms(data, pre_transforms, self.inverse_transforms())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/tasks/infer.py", line 288, in run_invert_transforms
    d = run_transforms(d, transforms, inverse=True, log_prefix="INV")
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/transform.py", line 80, in run_transforms
    data = t.inverse(data)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monai/transforms/spatial/dictionary.py", line 589, in inverse
    d[key] = inverse_transform(d[key])
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monai/transforms/spatial/array.py", line 447, in __call__
    raise ValueError(
ValueError: len(spatial_size) must be greater or equal to img spatial dimensions, got spatial_size=2 img=3.
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 132, in <module>
    run_main()
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/app.py", line 123, in run_main
    result = a.scoring(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 307, in scoring
    return task(copy.deepcopy(request), datastore if datastore else self.datastore())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/tasks/scoring/epistemic.py", line 126, in __call__
    model, model_ts = self._load_model(self.model, self.network)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/tasks/scoring/epistemic.py", line 117, in _load_model
    model.load_state_dict(model_state_dict)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1482, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DynUNet:
        Unexpected key(s) in state_dict: "deep_supervision_heads.0.conv.conv.weight", "deep_supervision_heads.0.conv.conv.bias", "deep_supervision_heads.1.conv.conv.weight", "deep_supervision_heads.1.conv.conv.bias", "deep_supervision_heads.2.conv.conv.weight", "deep_supervision_heads.2.conv.conv.bias", "deep_supervision_heads.3.conv.conv.weight", "deep_supervision_heads.3.conv.conv.bias".

[j-sieger]

@SachidanandAlle @diazandr3s Can i get some help on this ?

or please let me the know format of the data or any other specific things related to data to be taken care while preparing dataset for labelling.

I had DICOM images. i have converted them to NIFTI using python module dicom2nifti.dicom_series_to_nifti. Any suggestions on this ?

[diazandr3s]

Hi @j-sieger,

Are you using the deepedit_multilabel App? If yes, could you please do a clean installation of MONAI Label using monailabel-weekly. We've recently proposed a solution for the error you're getting ValueError: len(spatial_size) must be greater or equal to img spatial dimensions, got spatial_size=2 img=3.

For the other error Unexpected key(s) in state_dict: "deep_supervision_heads.0.conv.conv.weight" ... it seems you've changed the label_names dictionary or you're loading a pretrained model that doesn't match the network architecture. Could you provide more details of what changes you've done?

Please let us know.

[j-sieger]

@diazandr3s : i am using deepedit app as i want to label single class. but facing error as mentioned above when i click on fetch next images/running inference.

same deepedit app is working on my other datasets like Task02_Heart, Task09_Spleet.

I am facing same error like ValueError: len(spatial_size) must be greater or equal to img spatial dimensions, got spatial_size=2 img=3 when i tried on nifti dataset downloaded from kaggle.

[diazandr3s]

Thanks, @j-sieger. Did you install the latest MONAI Label version using monailabel-weekly? We've made a change recently to avoid that error. Please let us know.

[j-sieger]

@diazandr3s , yes i had monailabel-weekly==0.4.dev2207 and updated it to monailabel-weekly==0.4.dev2208 today. and other versions as follows.

monai                   0.8.1
monai-weekly            0.9.dev2208
monailabel              0.3.1
monailabel-weekly       0.4.dev2208

But i am facing same error when i click on next images from 3d slicer as it will run inference.

[2022-02-22 07:12:43,688] [17406] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - INV - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path', 'image_meta_dict', 'image_transforms', 'pred'])
[2022-02-22 07:12:43,702] [17406] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - INV - Transform (ToTensord): Time: 0.0137; image: (1, 256, 256, 128)(float32); pred: torch.Size([1, 256, 256, 128])(torch.float32)
[2022-02-22 07:12:43,708] [17406] [MainThread] [INFO] (uvicorn.access:442) -  - "POST /infer/deepedit_seg?image=output_file HTTP/1.1" 500
[2022-02-22 07:12:43,708] [17406] [MainThread] [ERROR] (uvicorn.error:380) - Exception in ASGI application
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 377, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/fastapi/applications.py", line 212, in __call__
    await super().__call__(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/endpoints/infer.py", line 177, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/endpoints/infer.py", line 160, in run_inference
    result = instance.infer(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/app.py", line 231, in infer
    result_file_name, result_json = task(request)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/tasks/infer.py", line 237, in __call__
    data = self.run_invert_transforms(data, pre_transforms, self.inverse_transforms())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/tasks/infer.py", line 288, in run_invert_transforms
    d = run_transforms(d, transforms, inverse=True, log_prefix="INV")
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monailabel/interfaces/utils/transform.py", line 80, in run_transforms
    data = t.inverse(data)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monai/transforms/spatial/dictionary.py", line 745, in inverse
    d[key] = inverse_transform(d[key])
  File "/home/ec2-user/.local/lib/python3.9/site-packages/monai/transforms/spatial/array.py", line 624, in __call__
    raise ValueError(
ValueError: len(spatial_size) must be greater or equal to img spatial dimensions, got spatial_size=2 img=3.
[2022-02-22 07:12:43,894] [17406] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:76) - Return code: 1
[2022-02-22 07:12:43,894] [17406] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:58) - COMMAND:: /opt/conda/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"dice","gpus":"all"}
[2022-02-22 07:12:44,042] [18617] [MainThread] [INFO] (__main__:38) - Initializing App from: /home/ec2-user/efs/apps/deepedit; studies: /home/ec2-user/efs/nnnn_Dataset/images/nifti; conf: {'epistemic_enabled': 'true'}

[diazandr3s]

Hi @j-sieger,

Please remove the SingleLabelSingleModalityd transform in both infer and train files.

This transform makes sure the images are single modality and labels and single class. However, this error is common when some headers are lost after converting from DICOM to NIFTI. I need to investigate more on this.

After removing those you should be able to perform inference. If not, use the ITK reader in the LoadImaged transform like this:

reader="itkreader"

For some reason, I kept thinking on the deepedit_multilabel and thought I did the same for the deepedit App. Sorry about this.

Please let me know how that goes :)

[j-sieger]

Hi @diazandr3s ,

It is working now. Thanks for the help.

[diazandr3s]

Glad to hear that! Sorry for the inconvenience. Closing issue!