search

Project-MONAI / MONAILabel

MONAI Label is an intelligent open source image labeling and learning tool.
https://docs.monai.io/projects/label
Apache License 2.0
591 stars 189 forks source link

RuntimeError: Unknown type: itkMatrixF44 #1064

Closed berombau closed 1 year ago

berombau commented 1 year ago

Describe the bug MONAILabel fails when running spleen_deepedit_annotation on in 3D Slicer 5.0.3 and with the MONAILabel plugin version: 7cd40d8 (2022-10-10).

To Reproduce Steps to reproduce the behavior:

  1. Run MONAI Label on the server: 
    
# install MONAI Label
pip install monailabel

download Bundle sample app to local directory

monailabel apps --name monaibundle --download --output apps

download a local study images, sample dataset such as spleen:

monailabel datasets --download --name Task09_Spleen --output datasets

monailabel start_server --app apps/monaibundle --studies datasets/Task09_Spleen/imagesTr --conf models "spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0"


2. On client, start 3D Slicer and MONAI label, input the server (e.g. http://cp0293:8000)
3. Click Active Learning > Next Sample
4. Click Auto Segmentation > Run for spleen_ct_segmentation_v0.3.1
5. Click SmartEdit / Deepgrow > Landmarks for spleen_deepedit_annotation_v0.3.0 and click on a spot in the image just outside of the automatic segmentation
6. Slicer error because of 500 Internal server error and an error in the server log:

(monai) [benjaminr@cp0293 MONAILabel]$ monailabel start_server --app workspace/monaibundle --studies datasets/Task09_Spleen/imagesTr --conf models "spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0" Using PYTHONPATH=/srv/scratch/benjaminr/anaconda3/envs::'/monaibundle/model/renalStructures_UNEST_segmentation_v0.2.0/scripts:'/monaibundle/model/renalStructures_UNEST_segmentation_v0.1.3/scripts:/srv/scratch/benjaminr/monai/MONAILabel/sample-apps/monaibundle/model/renalStructures_UNEST_segmentation_v0.1.3/scripts:/srv/scratch/benjaminr/monai/MONAILabel/sample-apps/monaibundle/model/renalStructures_UNEST_segmentation_v0.1.3/scripts

2022-10-12 16:43:45,007 - USING:: version = False 2022-10-12 16:43:45,007 - USING:: app = /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle 2022-10-12 16:43:45,007 - USING:: studies = /srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr 2022-10-12 16:43:45,007 - USING:: verbose = INFO 2022-10-12 16:43:45,007 - USING:: conf = [['models', 'spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0']] 2022-10-12 16:43:45,007 - USING:: host = 0.0.0.0 2022-10-12 16:43:45,007 - USING:: port = 8000 2022-10-12 16:43:45,007 - USING:: uvicorn_app = monailabel.app:app 2022-10-12 16:43:45,007 - USING:: ssl_keyfile = None 2022-10-12 16:43:45,007 - USING:: ssl_certfile = None 2022-10-12 16:43:45,007 - USING:: ssl_keyfile_password = None 2022-10-12 16:43:45,007 - USING:: ssl_ca_certs = None 2022-10-12 16:43:45,007 - USING:: workers = None 2022-10-12 16:43:45,007 - USING:: limit_concurrency = None 2022-10-12 16:43:45,007 - USING:: access_log = False 2022-10-12 16:43:45,007 - USING:: log_config = None 2022-10-12 16:43:45,007 - USING:: dryrun = False 2022-10-12 16:43:45,007 - USING:: action = start_server 2022-10-12 16:43:45,007 - ENV SETTINGS:: MONAI_LABEL_API_STR = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_PROJECT_NAME = MONAILabel 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_APP_DIR = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_STUDIES = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_AUTH_ENABLE = False 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_AUTH_DB = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_APP_CONF = '{}' 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_TRAIN = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_STRATEGY = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_SCORING = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_TASKS_BATCH_INFER = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_URL = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_USERNAME = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PASSWORD = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_API_KEY = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_CACHE_PATH = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_PROJECT = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_ASSET_PATH = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_DSA_ANNOTATION_GROUPS = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_USERNAME = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PASSWORD = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_PATH = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_QIDO_PREFIX = None 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_WADO_PREFIX = None 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_STOW_PREFIX = None 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_FETCH_BY_FRAME = False 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CONVERT_TO_NIFTI = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_SEARCH_FILTER = '{"Modality": "CT"}' 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_CACHE_EXPIRY = 180 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_PROXY_TIMEOUT = 30.0 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DICOMWEB_READ_TIMEOUT = 5.0 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_AUTO_RELOAD = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_DATASTORE_FILE_EXT = '[".nii.gz", ".nii", ".nrrd", ".jpg", ".png", ".tif", ".svs", ".xml"]' 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SERVER_PORT = 8000 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_CORS_ORIGINS = '[]' 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SESSIONS = True 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SESSION_PATH = 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_SESSION_EXPIRY = 3600 2022-10-12 16:43:45,008 - ENV SETTINGS:: MONAI_LABEL_INFER_CONCURRENCY = -1 2022-10-12 16:43:45,009 - ENV SETTINGS:: MONAI_LABEL_INFER_TIMEOUT = 600 2022-10-12 16:43:45,009 - ENV SETTINGS:: MONAI_LABEL_AUTO_UPDATE_SCORING = True 2022-10-12 16:43:45,009 - Allow Origins: [''] [2022-10-12 16:43:45,422] [49797] [MainThread] [INFO] (uvicorn.error:75) - Started server process [49797] [2022-10-12 16:43:45,423] [49797] [MainThread] [INFO] (uvicorn.error:45) - Waiting for application startup. [2022-10-12 16:43:45,423] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.app:38) - Initializing App from: /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle; studies: /srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr; conf: {'models': 'spleen_ct_segmentation_v0.3.1,spleen_deepedit_annotation_v0.3.0'} [2022-10-12 16:43:45,646] [49797] [MainThread] [INFO] (monailabel.utils.others.class_utils:37) - Subclass for MONAILabelApp Found: <class 'main.MyApp'> [2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (main:81) - +++ Adding Bundle from Zoo: spleen_ct_segmentation_v0.3.1 => {'checksum': '271cc5248be5b527b9a06f95e63cbbf91e81681e', 'source': 'https://github.com/Project-MONAI/model-zoo/releases/download/hosting_storage_v1/spleen_ct_segmentation_v0.3.1.zip'} => /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle/model/spleen_ct_segmentation_v0.3.1 [2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (main:81) - +++ Adding Bundle from Zoo: spleen_deepedit_annotation_v0.3.0 => {'checksum': '235ca2f1614f8ee189a309d7cefe1615d86a5e59', 'source': 'https://github.com/Project-MONAI/model-zoo/releases/download/hosting_storage_v1/spleen_deepedit_annotation_v0.3.0.zip'} => /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle/model/spleen_deepedit_annotation_v0.3.0 [2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (main:90) - +++ Using Models: ['spleen_ct_segmentation_v0.3.1', 'spleen_deepedit_annotation_v0.3.0'] [2022-10-12 16:43:45,751] [49797] [MainThread] [INFO] (monailabel.interfaces.app:129) - Init Datastore for: /srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr [2022-10-12 16:43:45,752] [49797] [MainThread] [INFO] (monailabel.datastore.local:128) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '*.xml'] [2022-10-12 16:43:45,762] [49797] [MainThread] [INFO] (monailabel.datastore.local:574) - Invalidate count: 0 [2022-10-12 16:43:45,763] [49797] [MainThread] [INFO] (monailabel.datastore.local:148) - Start observing external modifications on datastore (AUTO RELOAD) [2022-10-12 16:43:46,097] [49797] [MainThread] [INFO] (main:108) - +++ Adding Inferer:: spleen_ct_segmentation_v0.3.1 => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7fed02e8a4a0> [2022-10-12 16:43:46,769] [49797] [MainThread] [INFO] (main:108) - +++ Adding Inferer:: spleen_deepedit_annotation_v0.3.0 => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x7fed02e89990> [2022-10-12 16:43:46,770] [49797] [MainThread] [INFO] (main:122) - +++ Adding Trainer:: spleen_ct_segmentation_v0.3.1 => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7fed02e89960> [2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (main:122) - +++ Adding Trainer:: spleen_deepedit_annotation_v0.3.0 => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x7fed000793c0> [2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (main:132) - Active Learning Strategies:: ['random', 'first'] [2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: /home/benjaminr/.cache/monailabel/sessions [2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600 [2022-10-12 16:43:46,771] [49797] [MainThread] [INFO] (monailabel.interfaces.app:467) - App Init - completed [2022-10-12 16:43:46,773] [timeloop] [INFO] Starting Timeloop.. [2022-10-12 16:43:46,773] [49797] [MainThread] [INFO] (timeloop:60) - Starting Timeloop.. [2022-10-12 16:43:46,774] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fed02e9dd80> [2022-10-12 16:43:46,774] [49797] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x7fed02e9dd80> [2022-10-12 16:43:46,775] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set [2022-10-12 16:43:46,775] [49797] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set [2022-10-12 16:43:46,775] [49797] [MainThread] [INFO] (uvicorn.error:59) - Application startup complete. [2022-10-12 16:43:46,777] [49797] [MainThread] [INFO] (uvicorn.error:206) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) [2022-10-12 16:44:07,564] [49797] [MainThread] [INFO] (monailabel.endpoints.activelearning:43) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'} [2022-10-12 16:44:07,570] [49797] [MainThread] [INFO] (monailabel.tasks.activelearning.random:45) - Random: Selected Image: spleen_45; Weight: 1665585847 [2022-10-12 16:44:07,582] [49797] [MainThread] [INFO] (monailabel.endpoints.activelearning:59) - Next sample: {'id': 'spleen_45', 'weight': 1665585847, 'path': '/srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr/spleen_45.nii.gz', 'ts': 1665494588, 'name': 'spleen_45.nii.gz'} [2022-10-12 16:44:56,001] [49797] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'spleen_ct_segmentation_v0.3.1', 'image': 'spleen_45', 'device': 'cuda:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2022-10-12 16:44:56,002] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:258) - Infer Request (final): {'device': 'cuda:0', 'model': 'spleen_ct_segmentation_v0.3.1', 'image': '/srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr/spleen_45.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'A pre-trained model for volumetric (3D) segmentation of the spleen from CT image'} [2022-10-12 16:44:56,010] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Run Transform(s) [2022-10-12 16:44:56,010] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path'] [2022-10-12 16:44:57,789] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (LoadImaged): Time: 1.7768; image: (512, 512, 112)(torch.float32) [2022-10-12 16:44:57,790] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureChannelFirstd): Time: 0.0007; image: (1, 512, 512, 112)(torch.float32) [2022-10-12 16:44:57,795] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Orientationd): Time: 0.0042; image: (1, 512, 512, 112)(torch.float32) [2022-10-12 16:44:59,816] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (Spacingd): Time: 2.0205; image: (1, 328, 328, 223)(torch.float32) [2022-10-12 16:44:59,950] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (ScaleIntensityRanged): Time: 0.1329; image: (1, 328, 328, 223)(torch.float32) [2022-10-12 16:44:59,950] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - PRE - Transform (EnsureTyped): Time: 0.0002; image: (1, 328, 328, 223)(torch.float32) [2022-10-12 16:44:59,951] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:426) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': [96, 96, 96], 'sw_batch_size': 4, 'overlap': 0.5, 'mode': constant, 'sigma_scale': 0.125, 'padding_mode': constant, 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'roi_weight_map': None} [2022-10-12 16:44:59,952] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:375) - Infer model path: /srv/scratch/benjaminr/monai/MONAILabel/workspace/monaibundle/model/spleen_ct_segmentation_v0.3.1/models/model.pt [2022-10-12 16:45:07,481] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - POST - Run Transform(s) [2022-10-12 16:45:07,482] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'image_meta_dict', 'latencies', 'pred'] [2022-10-12 16:45:07,489] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Activationsd): Time: 0.0066; image: (1, 328, 328, 223)(torch.float32); pred: (2, 328, 328, 223)(torch.float32) [2022-10-12 16:45:08,324] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Invertd): Time: 0.834; image: (1, 328, 328, 223)(torch.float32); pred: (2, 512, 512, 112)(torch.float32) [2022-10-12 16:45:13,926] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (AsDiscreted): Time: 5.6012; image: (1, 328, 328, 223)(torch.float32); pred: (1, 512, 512, 112)(torch.float32) [2022-10-12 16:45:13,927] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:109) - POST - Transform (Restored): Time: 0.0006; image: (1, 328, 328, 223)(torch.float32); pred: (512, 512, 112)(torch.float32) [2022-10-12 16:45:13,927] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:458) - Writing Result... [2022-10-12 16:45:13,930] [49797] [MainThread] [INFO] (monailabel.transform.writer:188) - Result ext: .nrrd; write_to_file: True; dtype: uint8 [2022-10-12 16:45:19,251] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:290) - ++ Latencies => Total: 23.2501; Pre: 3.9487; Inferer: 7.5193; Invert: 0.0000; Post: 6.4573; Write: 5.3241 [2022-10-12 16:45:19,252] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:314) - Result File: /tmp/tmprudwjr3j.nrrd [2022-10-12 16:45:19,252] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:315) - Result Json Keys: ['label_names', 'latencies'] [2022-10-12 16:46:12,929] [49797] [MainThread] [INFO] (monailabel.endpoints.infer:160) - Infer Request: {'model': 'spleen_deepedit_annotation_v0.3.0', 'image': 'spleen_45', 'spleen': [[140, 256, 82]], 'label': 'spleen', 'device': 'cuda:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2022-10-12 16:46:12,932] [49797] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:258) - Infer Request (final): {'device': 'cuda:0', 'model': 'spleen_deepedit_annotation_v0.3.0', 'image': '/srv/scratch/benjaminr/monai/MONAILabel/datasets/Task09_Spleen/imagesTr/spleen_45.nii.gz', 'spleen': [[140, 256, 82]], 'label': 'spleen', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'This is a pre-trained model for 3D segmentation of the spleen organ from CT images using DeepEdit.'} [2022-10-12 16:46:12,947] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:63) - PRE - Run Transform(s) [2022-10-12 16:46:12,948] [49797] [MainThread] [INFO] (monailabel.interfaces.utils.transform:64) - PRE - Input Keys: ['device', 'model', 'image', 'spleen', 'label', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path'] [2022-10-12 16:46:14,737] [49797] [MainThread] [ERROR] (uvicorn.error:369) - Exception in ASGI application Traceback (most recent call last): File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi result = await app(self.scope, self.receive, self.send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call return await self.app(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/applications.py", line 269, in call await super().call(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/applications.py", line 124, in call await self.middleware_stack(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in call await self.app(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/exceptions.py", line 93, in call raise exc File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in call await self.app(scope, receive, sender) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/routing.py", line 670, in call await route.handle(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/routing.py", line 266, in handle await self.app(scope, receive, send) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/starlette/routing.py", line 65, in app response = await func(request) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app raw_response = await run_endpoint_function( File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function return await dependant.call(**values) File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/endpoints/infer.py", line 179, in api_run_inference return run_inference(background_tasks, model, image, session_id, params, file, label, output) File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/endpoints/infer.py", line 161, in run_inference result = instance.infer(request) File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/interfaces/app.py", line 299, in infer result_file_name, result_json = task(request) File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/tasks/infer/basic_infer.py", line 267, in call data = self.run_pre_transforms(data, pre_transforms) File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/tasks/infer/basic_infer.py", line 343, in run_pre_transforms return run_transforms(data, transforms, log_prefix="PRE", use_compose=False) File "/srv/scratch/benjaminr/monai/MONAILabel/monailabel/interfaces/utils/transform.py", line 93, in run_transforms data = t(data) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/dictionary.py", line 154, in call data = self._loader(d[key], reader) File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 271, in call meta_data = switch_endianness(meta_data, "<") File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 84, in switch_endianness data = {k: switch_endianness(v, new) for k, v in data.items()} File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 84, in data = {k: switch_endianness(v, new) for k, v in data.items()} File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 86, in switch_endianness raise RuntimeError(f"Unknown type: {type(data).name}") RuntimeError: Unknown type: itkMatrixF44


**Expected behavior**

After step 5, MONAI Label server should return a new segmentation where the old segmentation was grown to the region of the user click. 

**Environment**

Ensuring you use the relevant python executable, please paste the output of:

================================ Printing MONAI config...

MONAI version: 1.0.0 Numpy version: 1.23.2 Pytorch version: 1.12.1+cu113 MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False MONAI rev id: 170093375ce29267e45681fcec09dfa856e1d7e7 MONAI file: /srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/init.py

Optional dependencies: Pytorch Ignite version: 0.4.10 Nibabel version: 4.0.2 scikit-image version: 0.19.3 Pillow version: 9.2.0 Tensorboard version: 2.10.0 gdown version: 4.5.1 TorchVision version: 0.13.1+cu113 tqdm version: 4.64.1 lmdb version: 1.3.0 psutil version: 5.9.2 pandas version: NOT INSTALLED or UNKNOWN VERSION. einops version: 0.4.1 transformers version: NOT INSTALLED or UNKNOWN VERSION. mlflow version: NOT INSTALLED or UNKNOWN VERSION. pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================ Printing system config...

System: Linux Linux version: CentOS Linux 7 (Core) Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.17 Processor: x86_64 Machine: x86_64 Python version: 3.10.6 Process name: python Command: ['python', '-c', 'import monai; monai.config.print_debug_info()'] Open files: [popenfile(path='/home/benjaminr/.vscode-server/data/logs/20221012T155029/ptyhost.log', fd=19, position=2188, mode='a', flags=33793), popenfile(path='/home/benjaminr/.vscode-server/data/logs/20221012T155029/remoteagent.log', fd=20, position=1958, mode='a', flags=33793), popenfile(path='/home/benjaminr/.vscode-server/bin/74b1f979648cc44d385a2286793c226e611f59e7/vscode-remote-lock.benjaminr.74b1f979648cc44d385a2286793c226e611f59e7', fd=99, position=0, mode='w', flags=32769)] Num physical CPUs: 16 Num logical CPUs: 32 Num usable CPUs: 32 CPU usage (%): [90.8, 14.9, 17.0, 13.9, 13.0, 12.0, 13.4, 13.9, 82.1, 18.3, 17.8, 7.8, 14.6, 18.8, 6.0, 17.1, 7.8, 8.3, 6.9, 6.9, 10.0, 6.4, 5.5, 6.0, 6.0, 5.5, 16.7, 8.3, 5.5, 7.3, 100.0, 6.4] CPU freq. (MHz): 1515 Load avg. in last 1, 5, 15 mins (%): [10.5, 7.8, 6.5] Disk usage (%): 87.9 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 345.8 Available memory (GB): 285.6 Used memory (GB): 59.0

================================ Printing GPU config...

Num GPUs: 8 Has CUDA: True CUDA version: 11.3 cuDNN enabled: True cuDNN version: 8302 Current device: 0 Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86'] GPU 0 Name: NVIDIA A30 GPU 0 Is integrated: False GPU 0 Is multi GPU board: False GPU 0 Multi processor count: 56 GPU 0 Total memory (GB): 23.7 GPU 0 CUDA capability (maj.min): 8.0 GPU 1 Name: NVIDIA GeForce RTX 3090 GPU 1 Is integrated: False GPU 1 Is multi GPU board: False GPU 1 Multi processor count: 82 GPU 1 Total memory (GB): 23.7 GPU 1 CUDA capability (maj.min): 8.6 GPU 2 Name: NVIDIA TITAN V GPU 2 Is integrated: False GPU 2 Is multi GPU board: False GPU 2 Multi processor count: 80 GPU 2 Total memory (GB): 11.8 GPU 2 CUDA capability (maj.min): 7.0 GPU 3 Name: NVIDIA TITAN V GPU 3 Is integrated: False GPU 3 Is multi GPU board: False GPU 3 Multi processor count: 80 GPU 3 Total memory (GB): 11.8 GPU 3 CUDA capability (maj.min): 7.0 GPU 4 Name: NVIDIA GeForce RTX 3090 GPU 4 Is integrated: False GPU 4 Is multi GPU board: False GPU 4 Multi processor count: 82 GPU 4 Total memory (GB): 23.7 GPU 4 CUDA capability (maj.min): 8.6 GPU 5 Name: NVIDIA GeForce RTX 3090 GPU 5 Is integrated: False GPU 5 Is multi GPU board: False GPU 5 Multi processor count: 82 GPU 5 Total memory (GB): 23.7 GPU 5 CUDA capability (maj.min): 8.6 GPU 6 Name: NVIDIA GeForce RTX 3090 GPU 6 Is integrated: False GPU 6 Is multi GPU board: False GPU 6 Multi processor count: 82 GPU 6 Total memory (GB): 23.7 GPU 6 CUDA capability (maj.min): 8.6 GPU 7 Name: NVIDIA GeForce RTX 3090 GPU 7 Is integrated: False GPU 7 Is multi GPU board: False GPU 7 Multi processor count: 82 GPU 7 Total memory (GB): 23.7 GPU 7 CUDA capability (maj.min): 8.6



**Additional context**
Running 3D Slicer 5.0.3 r30893 / 7ea0f43 on a MacBook Pro (14-inch, 2021).
MONAILabel plugin version: 7cd40d8 (2022-10-10)
SachidanandAlle commented 1 year ago

I see segmentation model worked fine.. and failed for deepedit model.. is it all the time happening for every image.. i can't even say image issue because the same image is loaded by the same transform in first model but it fails on 2nd.. is it happening randomly?

SachidanandAlle commented 1 year ago

File "/srv/scratch/benjaminr/anaconda3/envs/monai/lib/python3.10/site-packages/monai/transforms/io/array.py", line 86, in switch_endianness raise RuntimeError(f"Unknown type: {type(data).__name__}")

I think there is some issue with 3.10 support.. not everything is compatible (itk or some other libraries).. can you please try with 3.7/3.8/3.9 python versions only

https://docs.monai.io/projects/label/en/latest/installation.html#install-monai-label

diazandr3s commented 1 year ago

Are you running the MONAI Label server on a MAC, @berombau?

tangy5 commented 1 year ago

@berombau Thanks for reporting this. I tried to reproduce the error with my local env. It works with my env, but as Sachi mentioned, it's not 3.10.6, it's 3.9.12. I also tried to create an env with python 3.10.6, but I can't build wheel with the "pip install monailabel". There might be env version compatible issue. Here is my working env, if this help you solve the problem:

================================
Printing MONAI config...
================================
MONAI version: 1.0.0
Numpy version: 1.21.5
Pytorch version: 1.12.0+cu102
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 170093375ce29267e45681fcec09dfa856e1d7e7
MONAI __file__: /home/yucheng/anaconda3/lib/python3.9/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: 0.4.10
Nibabel version: 3.1.1
scikit-image version: 0.19.2
Pillow version: 9.0.1
Tensorboard version: 2.9.1
gdown version: 4.5.1
TorchVision version: 0.13.0+cu102
tqdm version: 4.63.0
lmdb version: 1.3.0
psutil version: 5.8.0
pandas version: 1.4.2
einops version: 0.4.1
transformers version: NOT INSTALLED or UNKNOWN VERSION.
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: 0.4.3

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 20.04.4 LTS
Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.31
Processor: x86_64
Machine: x86_64
Python version: 3.9.12
Process name: ipython
Command: ['/home/yucheng/anaconda3/bin/python', '/home/yucheng/anaconda3/bin/ipython']
Open files: [popenfile(path='/home/yucheng/.ipython/profile_default/history.sqlite', fd=4, position=0, mode='r+', flags=688130), popenfile(path='/home/yucheng/.ipython/profile_default/history.sqlite', fd=5, position=0, mode='r+', flags=688130), popenfile(path='/home/yucheng/.ipython/profile_default/history.sqlite-journal', fd=30, position=0, mode='r+', flags=688130)]
Num physical CPUs: 4
Num logical CPUs: 8
Num usable CPUs: 8
CPU usage (%): [10.3, 10.9, 12.7, 11.8, 10.5, 11.0, 10.7, 37.8]
CPU freq. (MHz): 1702
Load avg. in last 1, 5, 15 mins (%): [23.8, 31.0, 33.6]
Disk usage (%): 33.5
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 31.2
Available memory (GB): 20.5
Used memory (GB): 9.6

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 10.2
cuDNN enabled: True
cuDNN version: 7605
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70']
GPU 0 Name: NVIDIA GeForce GTX 1080
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 20
GPU 0 Total memory (GB): 7.9
GPU 0 CUDA capability (maj.min): 6.1

Thanks

berombau commented 1 year ago

Thank you for your quick response, using python=3.9 worked! Here are the steps for reference. My setup is that the MONAI Label server runs on a CentOS workstation with GPUs and I access it over the network with 3D Slicer using a Mac laptop. 

mamba create -n monai -c pytorch python=3.9 pytorch torchvision torchaudio cudatoolkit=11.3 
conda activate monai
pip install monailabel
monailabel start_server ...

In my previous setup, I installed monailabel with 3.10 binaries using the itk pre-release:

pip install --upgrade --pre itk
diazandr3s commented 1 year ago

Thanks for letting us know, @berombau

wyli commented 1 year ago

the itk type won't be compatible with pytorch's data utils:

>>> import itk
>>> x = itk.Matrix[itk.F, 4, 4]()
>>> from torch.utils.data._utils.collate import default_collate
>>> default_collate([x,x])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/usr/local/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 151, in collate
    raise TypeError(default_collate_err_msg_format.format(elem_type))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'itk.itkMatrixPython.itkMatrixF44'>

I'll try to add a convert step