search

Project-MONAI / MONAILabel

MONAI Label is an intelligent open source image labeling and learning tool.
https://docs.monai.io/projects/label
Apache License 2.0
598 stars 194 forks source link

Cannot add multiple labels using MONAI Label server #616

Closed Suchi97 closed 2 years ago

Suchi97 commented 2 years ago

Describe the bug We are trying to do multi label annotation of liver and spleen. We used the spleen segmentation example and we followed this link to modify our multilabel annotation code. We changed the code as explained in the To Reproduce section,

To Reproduce Steps to reproduce the behavior:

  1. Go to apps/deepedit/main.py
    1. Change the out_channels in the dynunet network to 3. 0- background, 1- spleen, 2- liver 
      network_params = {
            "spatial_dims": 3,
            "in_channels": 3,
            "out_channels": 3,

  2. Go to apps/deepedit/train.py

    1. Add the sigmoid=true in loss function
    def loss_function(self, context: Context):
        return DiceLoss(sigmoid=True, squared_pred=True)
  3. Go to apps/deepedit/train.py
    1. Change the to_onehot value to 2. 
      def train_post_transforms(self, context: Context):
    return [
        Activationsd(keys="pred", sigmoid=True),
        AsDiscreted(keys="pred", threshold_values=True, to_onehot=3,logit_thresh=0.5),
    ]
  4. Then run the command monailabel start_server --app apps/deepedit --studies dataset/spleen

After running the above command, we selected an image by clicking the Next Sample button in Slicer App. The app pops up a dialog box showing Failed to run inference in MONAI Label server. In the terminal, we are getting the following error

Using PYTHONPATH=/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label:
[2022-02-07 16:05:38,689] [6221] [MainThread] [INFO] (__main__:277) - USING:: app = /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit
[2022-02-07 16:05:38,689] [6221] [MainThread] [INFO] (__main__:277) - USING:: studies = /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:38,689] [6221] [MainThread] [INFO] (__main__:277) - USING:: debug = False
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: conf = None
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: host = 0.0.0.0
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: port = 8000
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: log_config = None
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: dryrun = False
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:277) - USING:: action = start_server
[2022-02-07 16:05:38,690] [6221] [MainThread] [INFO] (__main__:278) -
[2022-02-07 16:05:40,396] [6221] [MainThread] [INFO] (uvicorn.error:64) - Started server process [6221]
[2022-02-07 16:05:40,396] [6221] [MainThread] [INFO] (uvicorn.error:26) - Waiting for application startup.
[2022-02-07 16:05:40,396] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:05:40,408] [6221] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:05:40,937] [6221] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:40,938] [6221] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:05:40,941] [6221] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:05:40,941] [6221] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:05:40,943] [6221] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:05:40,943] [6221] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:05:40,944] [6221] [MainThread] [INFO] (monailabel.interfaces.app:396) - App Init - completed
[2022-02-07 16:05:40,944] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'dice'}
[2022-02-07 16:05:40,945] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"dice","gpus":"all"}
[2022-02-07 16:05:40,945] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'sum'}
[2022-02-07 16:05:40,945] [timeloop] [INFO] Starting Timeloop..
[2022-02-07 16:05:40,945] [6221] [MainThread] [INFO] (timeloop:60) - Starting Timeloop..
[2022-02-07 16:05:40,964] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete.<locals>.run_schedulerat 0x7fdca9f8f790>
[2022-02-07 16:05:40,964] [6221] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete.<locals>.run_scheduler at 0x7fdca9f8f790>
[2022-02-07 16:05:40,965] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2022-02-07 16:05:40,965] [6221] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2022-02-07 16:05:40,965] [6221] [MainThread] [INFO] (uvicorn.error:38) - Application startup complete.
[2022-02-07 16:05:40,966] [6221] [MainThread] [INFO] (uvicorn.error:199) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2022-02-07 16:05:41,108] [6306] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:05:43,465] [6306] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:43,990] [6306] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:05:43,993] [6306] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:05:43,993] [6306] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:05:43,995] [6306] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:05:43,995] [6306] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:05:43,996] [6306] [MainThread] [INFO] (__main__:62) - Result: {}
[2022-02-07 16:05:44,262] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:05:44,263] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"sum","gpus":"all"}
[2022-02-07 16:05:44,404] [6391] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:05:46,738] [6391] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:05:47,294] [6391] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:05:47,298] [6391] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:05:47,298] [6391] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:05:47,300] [6391] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:05:47,300] [6391] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:05:48,962] [6391] [MainThread] [INFO] (__main__:62) - Result: {"spleen_22": {"sum": 1487375, "slices": 193}, "spleen_13": {"sum": 1000, "slices": 10}}
[2022-02-07 16:05:49,218] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:06:09,863] [6221] [MainThread] [INFO] (monailabel.endpoints.activelearning:42) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'}
[2022-02-07 16:06:09,864] [6221] [MainThread] [INFO] (monailabel.tasks.activelearning.random:44) - Random: Images: ['spleen_46', 'spleen_22', 'spleen_9', 'spleen_6', 'spleen_13']; Weight: [3515, 3403, 2772, 2388, 2491]
[2022-02-07 16:06:09,865] [6221] [MainThread] [INFO] (monailabel.tasks.activelearning.random:45) - Random: Selected Image:spleen_22
[2022-02-07 16:06:09,865] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'dice'}
[2022-02-07 16:06:09,866] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"dice","gpus":"all"}
[2022-02-07 16:06:09,867] [6221] [MainThread] [INFO] (monailabel.utils.async_tasks.task:36) - Scoring request: {'method': 'sum'}
[2022-02-07 16:06:09,897] [6221] [MainThread] [INFO] (monailabel.endpoints.activelearning:58) - Next sample: {'id': 'spleen_22', 'ts': 1643905192, 'checksum': 'SHA256:e2d12bf008afbbbcde50b3817b63e92fda3593999c944be0b52382b62d8cda9d', 'name': 'spleen_22.nii.gz', 'strategy': {'random': {'ts': 1644249969, 'client_id': 'user-xyz'}}, 'path': '/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_22.nii.gz'}
[2022-02-07 16:06:10,041] [6475] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:06:12,399] [6475] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:06:12,951] [6475] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:06:12,952] [6475] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:06:12,955] [6475] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:06:12,955] [6475] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:06:12,957] [6475] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:06:12,957] [6475] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:06:12,957] [6475] [MainThread] [INFO] (__main__:62) - Result: {}
[2022-02-07 16:06:13,225] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:06:13,226] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:57) - COMMAND:: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/bin/python -m monailabel.interfaces.utils.app -m scoring -r {"method":"sum","gpus":"all"}
[2022-02-07 16:06:13,389] [6559] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-07 16:06:14,636] [6221] [MainThread] [INFO] (monailabel.endpoints.infer:147) - Infer Request: {'model': 'deepedit_seg', 'image': 'spleen_22', 'client_id': 'user-xyz'}
[2022-02-07 16:06:14,637] [6221] [MainThread] [INFO] (monailabel.interfaces.app:224) - Image => /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_22.nii.gz
[2022-02-07 16:06:14,637] [6221] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:221) - Infer Request (final): {'model': 'deepedit_seg', 'image': '/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_22.nii.gz', 'client_id': 'user-xyz'}
[2022-02-07 16:06:14,640] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:61) - PRE - Run Transform
[2022-02-07 16:06:14,641] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - PRE - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path'])
[2022-02-07 16:06:15,714] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (LoadImaged): Time: 1.0716; image: (512, 512, 148)(float32)
[2022-02-07 16:06:15,714] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (SingleLabelSingleModalityd): Time: 0.0000; image: (512, 512, 148)(float32)
[2022-02-07 16:06:15,714] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (AddChanneld): Time: 0.0000; image: (1, 512, 512, 148)(float32)
[2022-02-07 16:06:15,783] [6559] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelAppFound: <class 'main.MyApp'>
[2022-02-07 16:06:16,370] [6559] [MainThread] [INFO] (main:104) - Pretrained Model Path: https://github.com/Project-MONAI/MONAILabel/releases/download/data/deepedit_dynunet_spleen.pt
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (main:109) - EPISTEMIC Enabled: 0; Samples: 5
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (main:113) - TTA Enabled: 0; Samples: 5
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (monailabel.interfaces.app:115) - Init Datastore for: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-07 16:06:16,371] [6559] [MainThread] [INFO] (monailabel.datastore.local:125) - Auto Reload: True; Extensions: ['*.nii.gz', '*.nii', '*.nrrd']
[2022-02-07 16:06:16,376] [6559] [MainThread] [INFO] (monailabel.datastore.local:534) - Invalidate count: 0
[2022-02-07 16:06:16,376] [6559] [MainThread] [INFO] (monailabel.datastore.local:145) - Start observing external modifications on datastore (AUTO RELOAD)
[2022-02-07 16:06:16,381] [6559] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /root/.cache/monailabel/sessions
[2022-02-07 16:06:16,381] [6559] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600
[2022-02-07 16:06:18,087] [6559] [MainThread] [INFO] (__main__:62) - Result: {"spleen_22": {"sum": 1487375, "slices": 193}, "spleen_13": {"sum": 1000, "slices": 10}}
[2022-02-07 16:06:18,366] [6221] [ThreadPoolExecutor-0_0] [INFO] (monailabel.utils.async_tasks.utils:75) - Return code: 0
[2022-02-07 16:06:19,511] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Spacingd): Time: 3.7970; image: (1, 496, 496, 222)(float32)
[2022-02-07 16:06:19,512] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Orientationd): Time: 0.0010; image: (1, 496, 496, 222)(float32)
[2022-02-07 16:06:20,061] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (NormalizeIntensityd): Time: 0.5483; image: (1, 496, 496, 222)(float32)
[2022-02-07 16:06:20,421] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Resized): Time: 0.3595; image: (1, 256, 256, 128)(float32)
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (DiscardAddGuidanced): Time: 0.0297; image: (3, 256, 256, 128)(float32)
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (ToTensord): Time: 0.0001; image: torch.Size([3, 256, 256, 128])(torch.float32)
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:344) - Running Inferer:: SimpleInferer
[2022-02-07 16:06:20,451] [6221] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:297) - Infer model path: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/trial_multilabel_monailabel_v3/deepedit/model/pretrained_dynunet.pt
[2022-02-07 16:06:24,198] [6221] [MainThread] [ERROR] (uvicorn.error:399) - Exception in ASGI application
Traceback (most recent call last):
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 396, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/cors.py", line 78, in __call__
    await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 580, in __call__
    await route.handle(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 241, in handle
    await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 52, in app
    response = await func(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/routing.py", line 214, in app
    raw_response = await run_endpoint_function(
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await dependant.call(**values)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 165, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/endpoints/infer.py", line 148, in run_inference
    result = instance.infer(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/app.py", line 225, in infer
    result_file_name, result_json = task(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 233, in __call__
    data = self.run_inferer(data, device=device)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 346, in run_inferer
    network = self._get_network(device)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/tasks/infer.py", line 322, in _get_network
    network.load_state_dict(model_state_dict, strict=self.load_strict)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1406, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DynUNet:
        size mismatch for input_block.conv1.conv.weight: copying a param with shape torch.Size([32, 3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 1, 3, 3, 3]).
        size mismatch for input_block.conv3.conv.weight: copying a param with shape torch.Size([32, 3, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 1, 1, 1, 1]).
        size mismatch for output_block.conv.conv.weight: copying a param with shape torch.Size([1, 32, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 32, 1, 1, 1]).
        size mismatch for output_block.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.0.conv.conv.weight: copying a param with shape torch.Size([1, 64, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1, 1]).
        size mismatch for deep_supervision_heads.0.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.1.conv.conv.weight: copying a param with shape torch.Size([1, 128, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 128, 1, 1, 1]).
        size mismatch for deep_supervision_heads.1.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.2.conv.conv.weight: copying a param with shape torch.Size([1, 256, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 256, 1, 1, 1]).
        size mismatch for deep_supervision_heads.2.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for deep_supervision_heads.3.conv.conv.weight: copying a param with shape torch.Size([1, 320, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 320, 1, 1, 1]).
        size mismatch for deep_supervision_heads.3.conv.conv.bias: copying a param with shape torch.Size([1]) from checkpoint, the shape in current model is torch.Size([3]).
        size mismatch for skip_layers.downsample.conv1.conv.weight: copying a param with shape torch.Size([32, 3, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 1, 3, 3, 3]).
        size mismatch for skip_layers.downsample.conv3.conv.weight: copying a param with shape torch.Size([32, 3, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 1, 1, 1, 1]).

Environment

MONAI Label version : 0.3.1

================================
Printing MONAI config...
================================
MONAI version: 0.8.0
Numpy version: 1.22.2
Pytorch version: 1.9.0+cu111
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: 714d00dffe6653e21260160666c4c201ab66511b

Optional dependencies:
Pytorch Ignite version: 0.4.6
Nibabel version: 3.2.1
scikit-image version: 0.19.1
Pillow version: 9.0.1
Tensorboard version: 2.8.0
gdown version: 4.2.1
TorchVision version: 0.10.0+cu111
tqdm version: 4.62.3
lmdb version: 1.3.0
psutil version: 5.9.0
pandas version: NOT INSTALLED or UNKNOWN VERSION.
einops version: NOT INSTALLED or UNKNOWN VERSION.
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.3 LTS
Platform: Linux-4.15.0-136-generic-x86_64-with-glibc2.10
Processor: x86_64
Machine: x86_64
Python version: 3.8.12
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 20
Num logical CPUs: 40
Num usable CPUs: 40
CPU usage (%): [9.8, 9.7, 9.1, 9.7, 9.1, 9.0, 9.0, 9.0, 10.5, 9.1, 9.1, 9.8, 9.1, 9.8, 8.4, 9.8, 9.1, 3.0, 9.7, 9.8, 9.7, 9.0, 9.1, 10.4, 9.1, 9.1, 9.8, 9.8, 9.8, 100.0, 11.9, 9.1, 9.1, 4.5, 9.8, 3.8, 9.8, 8.3, 8.4, 9.7]
CPU freq. (MHz): 2
Load avg. in last 1, 5, 15 mins (%): [0.1, 0.3, 0.6]
Disk usage (%): 76.9
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 251.8
Available memory (GB): 245.8
Used memory (GB): 4.0

================================
Printing GPU config...
================================
Num GPUs: 4
Has CUDA: True
CUDA version: 11.1
cuDNN enabled: True
cuDNN version: 8005
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']
GPU 0 Name: Tesla V100-DGXS-32GB
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 80
GPU 0 Total memory (GB): 31.7
GPU 0 CUDA capability (maj.min): 7.0
GPU 1 Name: Tesla V100-DGXS-32GB
GPU 1 Is integrated: False
GPU 1 Is multi GPU board: False
GPU 1 Multi processor count: 80
GPU 1 Total memory (GB): 31.7
GPU 1 CUDA capability (maj.min): 7.0
GPU 2 Name: Tesla V100-DGXS-32GB
GPU 2 Is integrated: False
GPU 2 Is multi GPU board: False
GPU 2 Multi processor count: 80
GPU 2 Total memory (GB): 31.7
GPU 2 CUDA capability (maj.min): 7.0
GPU 3 Name: Tesla V100-DGXS-32GB
GPU 3 Is integrated: False
GPU 3 Is multi GPU board: False
GPU 3 Multi processor count: 80
GPU 3 Total memory (GB): 31.7
GPU 3 CUDA capability (maj.min): 7.0

Any suggestions is appreciated. Thanks in advance

diazandr3s commented 2 years ago

Hi @Suchi97,

Thanks for raising this issue. We addressed the multilabel task in this App (https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/deepedit_multilabel). So please use the multilabel deepedit App instead.

If you'd like to work on segmenting the liver and spleen, please update this dictionary. Assign the segment numbers and names and then disable the use of pre-trained model (set to false) so you can train a model specifically for this task.

Hope this helps,

Please let us know

Suchi97 commented 2 years ago

Thanks @diazandr3s for your reply. I will try the above mentioned suggestions and let u know the results.

Suchi97 commented 2 years ago

Hi @diazandr3s

I tried the above steps,

  1. Used the deepedit_multilabel from the above mentioned link (https://github.com/Project-MONAI/MONAILabel/tree/main/sample-apps/deepedit_multilabel)

  2. Changed the dictionary values as below; self.label_names = { "spleen": 1, "liver": 2, "background": 0, }

    class MyApp(MONAILabelApp):
def __init__(self, app_dir, studies, conf):

    # Zero values are reserved to background. Non zero values are for the labels
    self.label_names = {
        "spleen": 1,
        "liver": 2,
        "background": 0,
    }

    network = conf.get("network", "dynunet")
  3. Changed the use_pretrained_model to false
        use_pretrained_model = strtobool(conf.get("use_pretrained_model", "false"))

But when I ran the monailabel start_server --app apps/deepedit_multilabel --studies dataset/spleen, I got the following error

Using PYTHONPATH=/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label:[2022-02-08 12:44:31,087] [26555] [MainThread] [INFO] (__main__:277) - USING:: app = /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/apps_org[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: studies = /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: debug = False
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: conf = None
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: host = 0.0.0.0
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: port = 8000
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: log_config = None
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: dryrun = False
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:277) - USING:: action = start_server
[2022-02-08 12:44:31,088] [26555] [MainThread] [INFO] (__main__:278) -
[2022-02-08 12:44:32,800] [26555] [MainThread] [INFO] (uvicorn.error:82) - Started server process [26555]
[2022-02-08 12:44:32,800] [26555] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup.
[2022-02-08 12:44:32,800] [26555] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/apps_org; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-08 12:44:32,810] [26555] [MainThread] [ERROR] (uvicorn.error:119) - Traceback (most recent call last):
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 621, in lifespan
    async with self.lifespan_context(app):
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 518, in __aenter__
    await self._router.startup()
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 598, in startup
    await handler()
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/app.py", line 99, in startup_event
    instance = app_instance()
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/interfaces/utils/app.py", line 44, in app_instance
    c = get_class_of_subclass_from_file("main", main_py, "MONAILabelApp")
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/utils/others/class_utils.py", line 53, in get_class_of_subclass_from_file
    return get_class_of_subclass(module_from_file(module_name, file_path), class_c)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/utils/others/class_utils.py", line 25, in module_from_file
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/apps_org/main.py", line 17, in <module>
    from lib import DeepEdit, DeepEditSeg, MyStrategy, MyTrain
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/apps_org/lib/__init__.py", line 14, in <module>
    from .train import MyTrain
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/apps_org/lib/train.py", line 34, in <module>
    from monailabel.deepedit.multilabel.transforms import (
ImportError: cannot import name 'AddInitialSeedPointMissingLabelsd' from 'monailabel.deepedit.multilabel.transforms' (/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/monailabel/deepedit/multilabel/transforms.py)

[2022-02-08 12:44:32,810] [26555] [MainThread] [ERROR] (uvicorn.error:56) - Application startup failed. Exiting.

Also, there was two other import issues also. They are FindAllValidSlicesMissingLabelsd , NormalizeLabelsInDatasetd. I solved these 3 import issues by changing AddInitialSeedPointMissingLabelsd to AddInitialSeedPointCustomd and FindAllValidSlicesMissingLabelsd to FindAllValidSlicesCustomd. But I couldn't find anything related to NormalizeLabelsInDatasetd. So I commented the same. If anything is wrong, Please correct me.

After changing the above packages, I could able to run the MONAI Label server using the command monailabel start_server --app apps/deepedit_multilabel --studies dataset/spleen

Next I opened the Slicer application and selected Active Learning as MONAI Label. Then clicked on the Next Sample. But the images looks like as below:

image

Also, I couldn't able to label the liver and spleen using scribbles. Can you please help me to use the scribbles for multi labels.

diazandr3s commented 2 years ago

Hi @Suchi97,

Thanks for the detailed report. Just yesterday I updated the multilabel deepedit (https://github.com/Project-MONAI/MONAILabel/pull/620) to support missing labels in the dataset.

Those new transforms are located here. If you're using the GitHub repository please do a pull, otherwise just copy those new transforms. Here is the NormalizeLabelsInDatasetd transform.

These are minor changes and you should still be able to train a model and make MONAI Label work as you already did.

The reason you're getting those results is that there is no a pre-trained model to segment only the liver and spleen. You should train a model first. For that, you can either create your own labels before training or use the BTCV dataset which already have those labels.

If you decided to use the BTCV dataset, just organize them and make sure images and labels have the same name as presented here

If you want to create your own labels using the Medical Segmentation Decathlon dataset (Spleen task), my suggestion is to use Grow From Seeds as presented here. Using scribbles for multilabel is still a work in progress as you can see in this issue

Hope this helps.

Please let us know

Suchi97 commented 2 years ago

Hi @diazandr3s ,

First of all thanks for your suggestions. We were able to make some progress using MONAI Label as per your comments.

But we are facing some issue. We tried to use UNETR as a pretrained model and used spleen, liver and background as labels. And we are getting the following error

[2022-02-16 19:34:47,278] [18978] [MainThread] [INFO] (monailabel.endpoints.activelearning:58) - Next sample: {'id': 'spleen_9', 'ts': 1644941818, 'checksum': 'SHA256:d85db2ef501f9a27826102b01f0d9f466c7cd32693ef202e6d557d3edb875ae6', 'name': 'spleen_9.nii.gz', 'path': '/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_9.nii.gz'}
[2022-02-16 19:34:47,522] [21162] [MainThread] [INFO] (__main__:38) - Initializing App from: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/sample-apps/deepedit_multilabel; studies: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen; conf: {}
[2022-02-16 19:34:49,925] [21162] [MainThread] [INFO] (monailabel.utils.others.class_utils:35) - Subclass for MONAILabelApp Found: <class 'main.MyApp'>
[2022-02-16 19:34:50,949] [18978] [MainThread] [INFO] (monailabel.endpoints.infer:159) - Infer Request: {'model': 'deepedit_seg', 'image': 'spleen_9', 'client_id': 'user-xyz'}
[2022-02-16 19:34:50,949] [18978] [MainThread] [INFO] (monailabel.interfaces.app:230) - Image => /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_9.nii.gz
[2022-02-16 19:34:50,950] [18978] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:221) - Infer Request (final): {'model': 'deepedit_seg', 'image': '/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/dataset/spleen/spleen_9.nii.gz', 'client_id': 'user-xyz'}
[2022-02-16 19:34:50,951] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:61) - PRE - Run Transform
[2022-02-16 19:34:50,951] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:62) - PRE - Input Keys: dict_keys(['model', 'image', 'client_id', 'image_path'])
[2022-02-16 19:34:51,309] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (LoadImaged): Time: 0.3569; image: (512, 512, 41)(float32)
[2022-02-16 19:34:51,309] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (AddChanneld): Time: 0.0000; image: (1, 512, 512, 41)(float32)
[2022-02-16 19:34:51,378] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Orientationd): Time: 0.0677; image: (1, 512, 512, 41)(float32)
[2022-02-16 19:34:51,425] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (ScaleIntensityRanged): Time: 0.0473; image: (1, 512, 512, 41)(float32)
[2022-02-16 19:34:51,522] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (Resized): Time: 0.0958; image: (1, 128, 128, 128)(float32)
[2022-02-16 19:34:51,537] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (DiscardAddGuidanced): Time: 0.0149; image: (4, 128, 128, 128)(float32)
[2022-02-16 19:34:51,537] [18978] [MainThread] [INFO] (monailabel.interfaces.utils.transform:98) - PRE - Transform (ToTensord): Time: 0.0001; image: torch.Size([4, 128, 128, 128])(torch.float32)
[2022-02-16 19:34:51,537] [18978] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:355) - Running Inferer:: SimpleInferer
[2022-02-16 19:34:51,537] [18978] [MainThread] [INFO] (monailabel.interfaces.tasks.infer:297) - Infer model path: /workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/sample-apps/deepedit_multilabel/model/pretrained_unetr.pt
[2022-02-16 19:34:52,390] [18978] [MainThread] [ERROR] (uvicorn.error:380) - Exception in ASGI application
Traceback (most recent call last):
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 377, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/applications.py", line 212, in __call__
    await super().__call__(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/routing.py", line 226, in app
    raw_response = await run_endpoint_function(
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
    return await dependant.call(**values)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/monailabel/endpoints/infer.py", line 177, in api_run_inference
    return run_inference(background_tasks, model, image, session_id, params, file, label, output)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/monailabel/endpoints/infer.py", line 160, in run_inference
    result = instance.infer(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/monailabel/interfaces/app.py", line 231, in infer
    result_file_name, result_json = task(request)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/monailabel/interfaces/tasks/infer.py", line 233, in __call__
    data = self.run_inferer(data, device=device)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/monailabel/interfaces/tasks/infer.py", line 360, in run_inferer
    network = self._get_network(device)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/UNETR_liver_trial/MONAILabel/monailabel/interfaces/tasks/infer.py", line 327, in _get_network
    network.load_state_dict(model_state_dict, strict=self.load_strict)
  File "/workspace/Hands_On_Lab_Ideation_HCLS_20thJan/Lab_Sessions/Lab_0_MONAI_Label/venv_label/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1406, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for UNETR:
        size mismatch for vit.patch_embedding.patch_embeddings.weight: copying a param with shape torch.Size([1536, 9, 16,16, 16]) from checkpoint, the shape in current model is torch.Size([1536, 4, 16, 16, 16]).
        size mismatch for encoder1.layer.conv1.conv.weight: copying a param with shape torch.Size([64, 9, 3, 3, 3]) from checkpoint, the shape in current model is torch.Size([64, 4, 3, 3, 3]).
        size mismatch for encoder1.layer.conv3.conv.weight: copying a param with shape torch.Size([64, 9, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([64, 4, 1, 1, 1]).
        size mismatch for out.conv.conv.weight: copying a param with shape torch.Size([8, 64, 1, 1, 1]) from checkpoint, the shape in current model is torch.Size([3, 64, 1, 1, 1]).
        size mismatch for out.conv.conv.bias: copying a param with shape torch.Size([8]) from checkpoint, the shape in current model is torch.Size([3]).

I believe that this issue is due to the change in the input channels and the labels. The original model has 8 labels and one image. For our model 3 labels and one image which can be see from the image above So before changing the labels, the input channels ("in_channels": len(self.label_names) + 1,) will be 9 according to the code. After changing the labels to "spleen", "liver" and background", the input channels become 4.

So how we should change the input channels inside the model similar to having custom channels for pretrained network. And similarly, the output channels are should also be adjusted. Looking in the above error, the model is not using the custom input and output channels.

Looking forward to your reply.

Thank you for help in advance

diazandr3s commented 2 years ago

Thanks for your message, @Suchi97

As you said, the problem is that the number of input channels is different. The current pre-trained model has 9 channels as input. The one you're trying to use has 4 channels (spleen, liver, background and image).

If you want a model for segmenting those organs, you either use the pre-trained model and then remove the other labels after doing inference, or you retrain a model specifically for those organs using the BTCV dataset. The good thing about MONAI Label is that it is rather easy to get a pre-trained model without much change. In this case, the only change is the dictionary before triggering the training.

I hope this helps. Please let us know how you proceed :)

Suchi97 commented 2 years ago

Thanks @diazandr3s for your reply

As per your suggestion, we retrained the model specifically for those labels (liver and spleen) using the BTCV dataset. Then used that trained model to label the liver and spleen, We could able to see some progress while doing the above steps. Thanks again for your help.

We are using GPU having 32 GB memory. So, when executing the MONAI label, the whole GPU memory is utilized. The result of the nvidia-smi is attached given below image

Do MONAI label needs 32 GB for doing the process? When we submit the label and then its starts the training process. The following error is obtained

RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 31.75 GiB total capacity; 510.83 MiB already allocated; 150.38 MiB free; 772.00 MiB reserved in total by PyTorch)

Can you please help me to solve this. Thanks in advance.

SachidanandAlle commented 2 years ago

if you are using multilabel.. and 512x512xN kind of higher spatial size.. with 32GB GPU.. this won't be enough.. you can switch to small spatial size.. please remember.. this is more of resource related choices and not related to application logic.

diazandr3s commented 2 years ago

Thanks, @Suchi97. Please confirm the image size you're using for this. I trained a model using a GPU of 24GB on images 128x128x128

Suchi97 commented 2 years ago

Thanks @SachidanandAlle and @diazandr3s for your reply. We trained the model using GPU of 32GB on images having the spatial size 128x128x128. Its working now.

diazandr3s commented 2 years ago

Glad to hear this. Thanks for letting us know.

Closing issue by now.