search

Project-MONAI / MONAILabel

MONAI Label is an intelligent open source image labeling and learning tool.
https://docs.monai.io/projects/label
Apache License 2.0
616 stars 195 forks source link

vertebra_pipeline errors #1770

Open linfanluntan opened 3 weeks ago

linfanluntan commented 3 weeks ago

Describe the bug I'm trying to do vertebra segmentation using the pre-trained models, localization_spine and localization_vertebra are working, however, vertebra_pipeline gives errors.

Server logs (monailabel02) c:>monailabel start_server --app MLapp008/radiology --studies ML_dataset/khoury --conf models all --conf bundles wholeBody_ct_segmentation,pancreas_ct_dints_segmentation --log_level debug Using PYTHONPATH=C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs;C:\Users\FullerLabAI\anaconda3\envs; "" C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\ignite\handlers\checkpoint.py:17: DeprecationWarning: TorchScript support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the torch.compile optimizer instead. from torch.distributed.optim import ZeroRedundancyOptimizer [2024-10-30 06:43:08,350] [28940] [MainThread] [INFO] (main:285) - USING:: version = False [2024-10-30 06:43:08,350] [28940] [MainThread] [INFO] (main:285) - USING:: app = C:\MLapp008\radiology [2024-10-30 06:43:08,352] [28940] [MainThread] [INFO] (main:285) - USING:: studies = C:\ML_dataset\khoury [2024-10-30 06:43:08,352] [28940] [MainThread] [INFO] (main:285) - USING:: verbose = INFO [2024-10-30 06:43:08,353] [28940] [MainThread] [INFO] (main:285) - USING:: conf = [['models', 'all'], ['bundles', 'wholeBody_ct_segmentation,pancreas_ct_dints_segmentation']] [2024-10-30 06:43:08,354] [28940] [MainThread] [INFO] (main:285) - USING:: host = 0.0.0.0 [2024-10-30 06:43:08,354] [28940] [MainThread] [INFO] (main:285) - USING:: port = 8000 [2024-10-30 06:43:08,355] [28940] [MainThread] [INFO] (main:285) - USING:: uvicorn_app = monailabel.app:app [2024-10-30 06:43:08,356] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile = None [2024-10-30 06:43:08,356] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_certfile = None [2024-10-30 06:43:08,357] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile_password = None [2024-10-30 06:43:08,357] [28940] [MainThread] [INFO] (main:285) - USING:: ssl_ca_certs = None [2024-10-30 06:43:08,358] [28940] [MainThread] [INFO] (main:285) - USING:: workers = None [2024-10-30 06:43:08,359] [28940] [MainThread] [INFO] (main:285) - USING:: limit_concurrency = None [2024-10-30 06:43:08,359] [28940] [MainThread] [INFO] (main:285) - USING:: access_log = False [2024-10-30 06:43:08,360] [28940] [MainThread] [INFO] (main:285) - USING:: root_path = [2024-10-30 06:43:08,361] [28940] [MainThread] [INFO] (main:285) - USING:: log_level = debug [2024-10-30 06:43:08,362] [28940] [MainThread] [INFO] (main:285) - USING:: log_config = None [2024-10-30 06:43:08,362] [28940] [MainThread] [INFO] (main:285) - USING:: dryrun = False [2024-10-30 06:43:08,363] [28940] [MainThread] [INFO] (main:285) - USING:: action = start_server [2024-10-30 06:43:08,364] [28940] [MainThread] [INFO] (main:296) - Allow Origins: [''] [2024-10-30 06:43:09,418] [28940] [MainThread] [INFO] (uvicorn.error:82) - Started server process [28940] [2024-10-30 06:43:09,418] [28940] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup. App Init... [2024-10-30 06:43:09,419] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: C:\MLapp008\radiology; studies: C:\ML_dataset\khoury; conf: {'models': 'all', 'bundles': 'wholeBody_ct_segmentation,pancreas_ct_dints_segmentation'} [2024-10-30 06:43:09,566] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'> [2024-10-30 06:43:09,576] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepedit.DeepEdit'> [2024-10-30 06:43:09,577] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_2d.Deepgrow2D'> [2024-10-30 06:43:09,577] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.deepgrow_3d.Deepgrow3D'> [2024-10-30 06:43:09,578] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_spine.LocalizationSpine'> [2024-10-30 06:43:09,579] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.localization_vertebra.LocalizationVertebra'> [2024-10-30 06:43:09,580] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation.Segmentation'> [2024-10-30 06:43:09,581] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_spleen.SegmentationSpleen'> [2024-10-30 06:43:09,582] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_vertebra.SegmentationVertebra'> [2024-10-30 06:43:09,582] [28940] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.sw_fastedit.SWFastEditConfig'> [2024-10-30 06:43:09,583] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: deepedit => lib.configs.deepedit.DeepEdit [2024-10-30 06:43:10,175] [28940] [MainThread] [INFO] (lib.configs.deepedit:141) - EPISTEMIC Enabled: False; Samples: 5 [2024-10-30 06:43:10,175] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: deepgrow_2d => lib.configs.deepgrow_2d.Deepgrow2D BasicUNet features: (32, 64, 128, 256, 512, 32). [2024-10-30 06:43:10,219] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: deepgrow_3d => lib.configs.deepgrow_3d.Deepgrow3D BasicUNet features: (32, 64, 128, 256, 512, 32). [2024-10-30 06:43:10,329] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: localization_spine => lib.configs.localization_spine.LocalizationSpine [2024-10-30 06:43:10,425] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: localization_vertebra => lib.configs.localization_vertebra.LocalizationVertebra [2024-10-30 06:43:10,520] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation => lib.configs.segmentation.Segmentation [2024-10-30 06:43:10,613] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation_spleen => lib.configs.segmentation_spleen.SegmentationSpleen [2024-10-30 06:43:10,644] [28940] [MainThread] [INFO] (lib.configs.segmentation_spleen:79) - EPISTEMIC Enabled: False; Samples: 5 [2024-10-30 06:43:10,644] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: segmentation_vertebra => lib.configs.segmentation_vertebra.SegmentationVertebra [2024-10-30 06:43:10,740] [28940] [MainThread] [INFO] (main:95) - +++ Adding Model: sw_fastedit => lib.configs.sw_fastedit.SWFastEditConfig {'tumor': 0, 'background': 1} Downloading from C:\MLapp008\radiology\model\pretrained_sw_fastedit.pt [2024-10-30 06:43:11,035] [28940] [MainThread] [INFO] (main:98) - +++ Using Models: ['deepedit', 'deepgrow_2d', 'deepgrow_3d', 'localization_spine', 'localization_vertebra', 'segmentation', 'segmentation_spleen', 'segmentation_vertebra', 'sw_fastedit'] [2024-10-30 06:43:11,419] [28940] [MainThread] [INFO] (monailabel.utils.others.generic:304) - +++ Adding Bundle from Zoo: wholeBody_ct_segmentation => C:\MLapp008\radiology\model\wholeBody_ct_segmentation [2024-10-30 06:43:11,420] [28940] [MainThread] [INFO] (monailabel.utils.others.generic:304) - +++ Adding Bundle from Zoo: pancreas_ct_dints_segmentation => C:\MLapp008\radiology\model\pancreas_ct_dints_segmentation [2024-10-30 06:43:11,421] [28940] [MainThread] [INFO] (monailabel.utils.others.generic:364) - +++ Using Bundle Models: ['wholeBody_ct_segmentation', 'pancreas_ct_dints_segmentation'] [2024-10-30 06:43:11,422] [28940] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: C:\ML_dataset\khoury [2024-10-30 06:43:11,423] [28940] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '*.xml'] [2024-10-30 06:43:11,437] [28940] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0 [2024-10-30 06:43:11,437] [28940] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD) [2024-10-30 06:43:11,447] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepedit => <lib.infers.deepedit.DeepEdit object at 0x000001FC2DD5F070> [2024-10-30 06:43:11,447] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepedit_seg => <lib.infers.deepedit.DeepEdit object at 0x000001FC2E7AFD30> [2024-10-30 06:43:11,448] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepgrow_2d => <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E837520> [2024-10-30 06:43:11,448] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: deepgrow_3d => <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E8375E0> [2024-10-30 06:43:11,448] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: localization_spine => <lib.infers.localization_spine.LocalizationSpine object at 0x000001FC2E837880> [2024-10-30 06:43:11,449] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: localization_vertebra => <lib.infers.localization_vertebra.LocalizationVertebra object at 0x000001FC2E837430> [2024-10-30 06:43:11,449] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation => <lib.infers.segmentation.Segmentation object at 0x000001FC2E837040> [2024-10-30 06:43:11,449] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation_spleen => <lib.infers.segmentation_spleen.SegmentationSpleen object at 0x000001FC2E8372B0> [2024-10-30 06:43:11,450] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: segmentation_vertebra => <lib.infers.segmentation_vertebra.SegmentationVertebra object at 0x000001FC2E837250> [2024-10-30 06:43:11,460] [28940] [MainThread] [INFO] (main:128) - +++ Adding Inferer:: sw_fastedit => <lib.infers.sw_fastedit.SWFastEdit object at 0x000001FC2E837160> [2024-10-30 06:43:11,577] [28940] [MainThread] [INFO] (main:137) - +++ Adding Bundle Inferer:: wholeBody_ct_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC3914F490> You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. 2024-10-30 06:43:11.609109 - Length of input patch is recommended to be a multiple of 32. [2024-10-30 06:43:13,635] [28940] [MainThread] [INFO] (main:137) - +++ Adding Bundle Inferer:: pancreas_ct_dints_segmentation => <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC2E837220> [2024-10-30 06:43:13,636] [28940] [MainThread] [INFO] (main:194) - {'deepedit': <lib.infers.deepedit.DeepEdit object at 0x000001FC2DD5F070>, 'deepedit_seg': <lib.infers.deepedit.DeepEdit object at 0x000001FC2E7AFD30>, 'deepgrow_2d': <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E837520>, 'deepgrow_3d': <lib.infers.deepgrow.Deepgrow object at 0x000001FC2E8375E0>, 'localization_spine': <lib.infers.localization_spine.LocalizationSpine object at 0x000001FC2E837880>, 'localization_vertebra': <lib.infers.localization_vertebra.LocalizationVertebra object at 0x000001FC2E837430>, 'segmentation': <lib.infers.segmentation.Segmentation object at 0x000001FC2E837040>, 'segmentation_spleen': <lib.infers.segmentation_spleen.SegmentationSpleen object at 0x000001FC2E8372B0>, 'segmentation_vertebra': <lib.infers.segmentation_vertebra.SegmentationVertebra object at 0x000001FC2E837250>, 'sw_fastedit': <lib.infers.sw_fastedit.SWFastEdit object at 0x000001FC2E837160>, 'wholeBody_ct_segmentation': <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC3914F490>, 'pancreas_ct_dints_segmentation': <monailabel.tasks.infer.bundle.BundleInferTask object at 0x000001FC2E837220>, 'Histogram+GraphCut': <monailabel.scribbles.infer.HistogramBasedGraphCut object at 0x000001FC2EC3C190>, 'GMM+GraphCut': <monailabel.scribbles.infer.GMMBasedGraphCut object at 0x000001FC2EB94610>, 'deepgrow_pipeline': <lib.infers.deepgrow_pipeline.InferDeepgrowPipeline object at 0x000001FC2EB945E0>, 'vertebra_pipeline': <lib.infers.vertebra_pipeline.InferVertebraPipeline object at 0x000001FC2EB945B0>} [2024-10-30 06:43:13,637] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: deepedit => <lib.trainers.deepedit.DeepEdit object at 0x000001FC2EB946D0> [2024-10-30 06:43:13,638] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: deepgrow_2d => <lib.trainers.deepgrow.Deepgrow object at 0x000001FC2EB94700> [2024-10-30 06:43:13,639] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: deepgrow_3d => <lib.trainers.deepgrow.Deepgrow object at 0x000001FC2EB94730> [2024-10-30 06:43:13,639] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: localization_spine => <lib.trainers.localization_spine.LocalizationSpine object at 0x000001FC2EB94760> [2024-10-30 06:43:13,640] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: localization_vertebra => <lib.trainers.localization_vertebra.LocalizationVertebra object at 0x000001FC2EB94790> [2024-10-30 06:43:13,640] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation => <lib.trainers.segmentation.Segmentation object at 0x000001FC2EB947C0> [2024-10-30 06:43:13,641] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation_spleen => <lib.trainers.segmentation_spleen.SegmentationSpleen object at 0x000001FC2EB947F0> [2024-10-30 06:43:13,642] [28940] [MainThread] [INFO] (main:209) - +++ Adding Trainer:: segmentation_vertebra => <lib.trainers.segmentation_vertebra.SegmentationVertebra object at 0x000001FC2EB94820> [2024-10-30 06:43:13,644] [28940] [MainThread] [INFO] (main:221) - +++ Adding Bundle Trainer:: wholeBody_ct_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x000001FC2EB94670> [2024-10-30 06:43:13,668] [28940] [MainThread] [INFO] (main:221) - +++ Adding Bundle Trainer:: pancreas_ct_dints_segmentation => <monailabel.tasks.train.bundle.BundleTrainTask object at 0x000001FC2EB94640> [2024-10-30 06:43:13,669] [28940] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\FullerLabAI.cache\monailabel\sessions [2024-10-30 06:43:13,669] [28940] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600 [2024-10-30 06:43:13,670] [28940] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed [2024-10-30 06:43:13,670] [timeloop] [INFO] Starting Timeloop.. [2024-10-30 06:43:13,670] [28940] [MainThread] [INFO] (timeloop:60) - Starting Timeloop.. [2024-10-30 06:43:13,672] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000001FC2B8D1F70> [2024-10-30 06:43:13,672] [28940] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000001FC2B8D1F70> [2024-10-30 06:43:13,672] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set [2024-10-30 06:43:13,672] [28940] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set [2024-10-30 06:43:13,673] [28940] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete. [2024-10-30 06:43:13,673] [28940] [MainThread] [INFO] (uvicorn.error:214) - Uvicorn running on http://0.0.0.0:8000/ (Press CTRL+C to quit) [2024-10-30 06:43:35,581] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'} [2024-10-30 06:43:35,582] [28940] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: Output Volume1; Weight: 582 [2024-10-30 06:43:35,587] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': 'Output Volume1', 'weight': 582, 'path': 'C:\ML_dataset\khoury\Output Volume1.nii.gz', 'ts': 1729741578, 'name': 'Output Volume1.nii.gz', 'strategy': {'random': {'ts': 1730288615, 'client_id': 'user-xyz'}}} [2024-10-30 06:43:44,811] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'random', 'client_id': 'user-xyz'} [2024-10-30 06:43:44,812] [28940] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: spleen_53; Weight: 591 [2024-10-30 06:43:44,816] [28940] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': 'spleen_53', 'weight': 591, 'path': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'ts': 1729716311, 'name': 'spleen_53.nii.gz', 'strategy': {'random': {'ts': 1730288624, 'client_id': 'user-xyz'}}} [2024-10-30 06:43:57,815] [28940] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'vertebra_pipeline', 'image': 'spleen_53', 'device': 'NVIDIA RTX A6000:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2024-10-30 06:43:57,816] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda:0', 'model': 'vertebra_pipeline', 'image': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'Combines three stage for vertebra segmentation', 'image_path': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'pipeline_mode': True} [2024-10-30 06:43:57,818] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-10-30 06:43:57,819] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode'] [2024-10-30 06:44:05,293] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 7.472; image: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 06:44:05,371] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.078; image: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 06:44:05,372] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 06:44:05,375] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.002; image: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 06:44:06,471] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 1.096; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:06,472] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:07,414] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.941; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:07,478] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.063; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:07,478] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-10-30 06:44:07,480] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_spine.pt You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. [2024-10-30 06:44:14,611] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-10-30 06:44:14,611] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode', 'latencies', 'image_cached', 'pred'] [2024-10-30 06:44:14,613] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32) [2024-10-30 06:44:14,622] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.009; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32) [2024-10-30 06:44:14,627] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.005; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:19,571] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 4.943; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:19,572] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (BinaryMaskd): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 06:44:19,574] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 06:44:19,574] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 21.7583; Pre: 9.6610; Inferer: 7.1322; Invert: 0.0000; Post: 4.9640; Write: 0.0000 [2024-10-30 06:44:19,575] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids'] [2024-10-30 06:44:19,576] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:24) - **** DATA **** [2024-10-30 06:44:19,577] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: device = cuda:0 [2024-10-30 06:44:19,577] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: model = vertebra_pipeline [2024-10-30 06:44:19,578] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image = torch.Size([1, 512, 512, 156]) [2024-10-30 06:44:19,578] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_extension = .nrrd [2024-10-30 06:44:19,579] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_dtype = uint8 [2024-10-30 06:44:19,580] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: client_id = user-xyz [2024-10-30 06:44:19,580] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: description = Combines three stage for vertebra segmentation [2024-10-30 06:44:19,581] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image_path = C:\ML_dataset\khoury\spleen_53.nii.gz [2024-10-30 06:44:19,582] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: label = torch.Size([512, 512, 156]) [2024-10-30 06:44:19,584] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: pipeline_mode = True [2024-10-30 06:44:19,585] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:38) - ** monai.transforms.croppad.dictionary CropForegroundd.init:allow_smaller: Current default value of argument allow_smaller=True has been deprecated since version 1.2. It will be changed to allow_smaller=False in version 1.5. [2024-10-30 06:44:19,586] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-10-30 06:44:19,587] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode'] [2024-10-30 06:44:19,588] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 06:44:19,589] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 06:44:19,881] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.292; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 06:44:19,882] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 06:44:19,993] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.11; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 06:44:20,030] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.036; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 06:44:20,038] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CropForegroundd): Time: 0.008; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32) [2024-10-30 06:44:20,039] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': device(type='cpu'), 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-10-30 06:44:20,039] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_vertebra.pt [2024-10-30 06:44:20,717] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s) [2024-10-30 06:44:20,717] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred'] [2024-10-30 06:44:21,064] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (CropForegroundd): Time: 0.346; image: torch.Size([25, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32) [2024-10-30 06:45:01,703] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Spacingd): Time: 40.6367; image: torch.Size([25, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32) [2024-10-30 06:45:01,704] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-10-30 06:45:01,705] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred'] [2024-10-30 06:45:01,706] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32) [2024-10-30 06:45:02,111] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.404; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32) [2024-10-30 06:45:03,797] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 1.686; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 06:45:10,570] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 6.772; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 06:45:10,571] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 06:45:11,412] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (VertebraLocalizationSegmentation): Time: 0.838; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 06:45:11,412] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 51.8357; Pre: 0.4530; Inferer: 0.6380; Invert: 41.0257; Post: 9.7090; Write: 0.0000 [2024-10-30 06:45:11,413] [28940] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['centroids', 'label_names', 'latencies'] 0it [00:00, ?it/s] [2024-10-30 06:45:11,416] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST(P) - Run Transform(s) [2024-10-30 06:45:11,416] [28940] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST(P) - Input Keys: ['model', 'image', 'device', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pred'] [2024-10-30 06:45:11,417] [28940] [MainThread] [ERROR] (uvicorn.error:411) - Exception in ASGI application Traceback (most recent call last): File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 406, in run_asgi result = await app( # type: ignore[func-returns-value] File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 60, in call return await self.app(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\fastapi\applications.py", line 1054, in call await super().call(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\applications.py", line 113, in call await self.middleware_stack(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\errors.py", line 187, in call raise exc File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\errors.py", line 165, in call await self.app(scope, receive, _send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\cors.py", line 85, in call await self.app(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\middleware\exceptions.py", line 62, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app raise exc File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 715, in call await self.middleware_stack(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 735, in app await route.handle(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 288, in handle await self.app(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app raise exc File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\starlette\routing.py", line 73, in app response = await f(request) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\fastapi\routing.py", line 301, in app raw_response = await run_endpoint_function( File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\fastapi\routing.py", line 212, in run_endpoint_function return await dependant.call(**values) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\endpoints\infer.py", line 206, in api_run_inference return run_inference(background_tasks, model, image, session_id, params, file, label, output) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\endpoints\infer.py", line 172, in run_inference result = instance.infer(request) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\interfaces\app.py", line 307, in infer result_file_name, result_json = task(request) File "C:\MLapp008\radiology\lib\infers\vertebra_pipeline.py", line 158, in call data = run_transforms(data, [Restored(keys="pred", ref_image="image")], log_prefix="POST(P)", use_compose=False) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\interfaces\utils\transform.py", line 106, in run_transforms data = t(data) File "C:\Users\FullerLabAI\anaconda3\envs\monailabel02\lib\site-packages\monailabel\transform\post.py", line 127, in call current_size = result.shape[1:] if self.has_channel else result.shape AttributeError: 'NoneType' object has no attribute 'shape'

To Reproduce Steps to reproduce the behavior:

conda create -n monailabel02 python=3.9 conda activate monailabel02 monailabel apps --download --name radiology --output MLapp008 monailabel apps --download --name monaibundle --output MLapp008 monailabel start_server --app MLapp008/radiology --studies ML_dataset/khoury --conf models all --conf bundles wholeBody_ct_segmentation,pancreas_ct_dints_segmentation or: monailabel start_server --app MLapp008/radiology --studies ML_dataset/khoury --conf models all --conf bundles wholeBody_ct_segmentation,pancreas_ct_dints_segmentation --log_level debug run Monailabel in 3D slicer run vertebra_pipeline in auto segmentation

linfanluntan commented 3 weeks ago

I modified vertebra_pipeline.py

def call(self, request): start = time.time() request.update({"image_path": request.get("image")})

device = name_to_device(request.get("device", "cuda"))
request["device"] = device

try:
    # Run first stage
    logging.info("Starting locate_spine stage")
    d1, r1, l1 = self.locate_spine(request)
    image = d1["image"]
    label = d1["pred"]
    logging.info(f"locate_spine completed. Image shape: {image.shape}, Label shape: {label.shape}")

    # Run second stage
    logging.info("Starting locate_vertebra stage")
    d2, r2, l2 = self.locate_vertebra(request, image, label)
    centroids = r2["centroids"]
    logging.info(f"locate_vertebra completed. Centroids: {centroids}")

    # Run third stage
    logging.info("Starting segment_vertebra stage")
    result_mask, l3 = self.segment_vertebra(request, image, centroids)
    logging.info(f"segment_vertebra completed. Result mask shape: {result_mask.shape if result_mask is not None else 'None'}")

    if result_mask is None:
        raise ValueError("segment_vertebra returned None result_mask")

    # Finalize the mask/result
    data = copy.deepcopy(request)
    data.update({"pred": result_mask, "image": image})

    logging.info("Starting post-processing")
    data = run_transforms(data, [Restored(keys="pred", ref_image="image")], log_prefix="POST(P)", use_compose=False)

    if "pred" not in data or data["pred"] is None:
        raise ValueError("Post-processing resulted in None prediction")

    begin = time.time()
    result_file, _ = Writer(label="pred")(data)
    latency_write = round(time.time() - begin, 2)

    total_latency = round(time.time() - start, 2)
    result_json = {
        "label_names": self.task_seg_vertebra.labels,
        "centroids": centroids,
        "latencies": {
            "locate_spine": l1,
            "locate_vertebra": l2,
            "segment_vertebra": l3,
            "write": latency_write,
            "total": total_latency,
        },
    }

    logging.info(f"Result Mask (aggregated/pre-restore): {result_mask.shape}; total_latency: {total_latency}")
    return result_file, result_json

except Exception as e:
    logging.error(f"Error in vertebra pipeline: {str(e)}")
    raise

After running, we got: File "C:\MLapp008\radiology\lib\infers\vertebra_pipeline.py", line 162, in call raise ValueError("segment_vertebra returned None result_mask") ValueError: segment_vertebra returned None result_mask

linfanluntan commented 3 weeks ago

modified segment_vertebra

def segment_vertebra(self, request, image, centroids):
    original_size = list(image.shape)
    result_mask = None
    l = None
    image_cached = None
    count = 0

    logging.info(f"Starting vertebra segmentation for {len(centroids)} centroids")
    logging.info(f"Initial memory usage: {psutil.virtual_memory().percent}%")

    try:
        for centroid in tqdm(centroids):
            logging.info(f"Processing centroid: {centroid}")

            req = copy.deepcopy(request)
            req.update({
                "image": image,
                "image_cached": image_cached,
                "original_size": original_size,
                "centroids": [centroid],
                "pipeline_mode": True,
                "logging": "ERROR" if count > 1 else "INFO",
            })

            d, r = self.task_seg_vertebra(req)
            l = self._latencies(r, l)

            if "image" not in d or d["image"] is None:
                raise ValueError(f"Segmentation failed for centroid {centroid}")

            image = d["image"]
            image_cached = image

            # Paste/Merge each mask
            v = d["current_label"]
            s = d["slices_cropped"]
            m = d["pred"]

            if m is None:
                logging.warning(f"Null prediction for centroid {centroid}")
                continue

            m[m > 0] = v

            mask = torch.zeros_like(image)
            mask[:, s[-3][0]:s[-3][1], s[-2][0]:s[-2][1], s[-1][0]:s[-1][1]] = m

            if result_mask is None:
                result_mask = mask
            else:
                result_mask = result_mask + mask
                result_mask[result_mask > v] = v

            count += 1

            if count % 5 == 0:  # Log memory usage every 5 iterations
                logging.info(f"Memory usage after {count} centroids: {psutil.virtual_memory().percent}%")

        if result_mask is None:
            raise ValueError("No valid segmentation produced")

        logging.info(f"Vertebra segmentation completed. Result mask shape: {result_mask.shape}")
        return result_mask, l

    except Exception as e:
        logging.error(f"Error in vertebra segmentation: {str(e)}")
        raise

    finally:
        logging.info(f"Final memory usage: {psutil.virtual_memory().percent}%")

[2024-10-30 09:53:23,907] [26228] [MainThread] [INFO] (monailabel.endpoints.infer:171) - Infer Request: {'model': 'vertebra_pipeline', 'image': 'spleen_53', 'device': 'NVIDIA RTX A6000:0', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz'} [2024-10-30 09:53:23,908] [26228] [MainThread] [INFO] (root:166) - Starting locate_spine stage [2024-10-30 09:53:23,909] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:280) - Infer Request (final): {'device': 'cuda:0', 'model': 'vertebra_pipeline', 'image': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'result_extension': '.nrrd', 'result_dtype': 'uint8', 'client_id': 'user-xyz', 'description': 'Combines three stage for vertebra segmentation', 'image_path': 'C:\ML_dataset\khoury\spleen_53.nii.gz', 'pipeline_mode': True} [2024-10-30 09:53:23,910] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-10-30 09:53:23,910] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode'] [2024-10-30 09:53:28,391] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImaged): Time: 4.48; image: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 09:53:28,541] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureTyped): Time: 0.15; image: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 09:53:28,542] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 09:53:28,543] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.001; image: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 09:53:28,567] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.024; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:28,568] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:28,808] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.24; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:28,847] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.038; image: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:28,847] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': None, 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-10-30 09:53:28,848] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_spine.pt [2024-10-30 09:53:35,816] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-10-30 09:53:35,816] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'pipeline_mode', 'latencies', 'image_cached', 'pred'] [2024-10-30 09:53:35,817] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32) [2024-10-30 09:53:35,826] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.008; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([25, 330, 330, 359])(torch.float32) [2024-10-30 09:53:35,827] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:40,819] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 4.992; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:40,820] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (BinaryMaskd): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([1, 330, 330, 359])(torch.float32) [2024-10-30 09:53:40,821] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0; image: torch.Size([1, 330, 330, 359])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 16.9131; Pre: 4.9370; Inferer: 6.9682; Invert: 0.0000; Post: 5.0070; Write: 0.0000 [2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['label_names', 'latencies', 'centroids'] [2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (root:170) - locate_spine completed. Image shape: torch.Size([1, 512, 512, 156]), Label shape: torch.Size([512, 512, 156]) [2024-10-30 09:53:40,822] [26228] [MainThread] [INFO] (root:173) - Starting locate_vertebra stage [2024-10-30 09:53:40,823] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:24) - **** DATA **** [2024-10-30 09:53:40,823] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: device = cuda:0 [2024-10-30 09:53:40,823] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: model = vertebra_pipeline [2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image = torch.Size([1, 512, 512, 156]) [2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_extension = .nrrd [2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: result_dtype = uint8 [2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: client_id = user-xyz [2024-10-30 09:53:40,824] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: description = Combines three stage for vertebra segmentation [2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: image_path = C:\ML_dataset\khoury\spleen_53.nii.gz [2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: label = torch.Size([512, 512, 156]) [2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:27) - Data key: pipeline_mode = True [2024-10-30 09:53:40,825] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:38) - ** [2024-10-30 09:53:40,826] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2024-10-30 09:53:40,826] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode'] [2024-10-30 09:53:40,827] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 09:53:40,827] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CacheObjectd): Time: 0.0; image: torch.Size([1, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 09:53:41,185] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (Spacingd): Time: 0.357; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 09:53:41,186] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 09:53:41,297] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (GaussianSmoothd): Time: 0.111; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 09:53:41,334] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityd): Time: 0.036; image: torch.Size([1, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32) [2024-10-30 09:53:41,342] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CropForegroundd): Time: 0.008; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32) [2024-10-30 09:53:41,342] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:503) - Inferer:: cuda:0 => SlidingWindowInferer => {'roi_size': (96, 96, 96), 'sw_batch_size': 2, 'overlap': 0.4, 'mode': gaussian, 'sigma_scale': 0.125, 'padding_mode': 'replicate', 'cval': 0.0, 'sw_device': None, 'device': device(type='cpu'), 'progress': False, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'with_coord': False, 'roi_weight_map': None} [2024-10-30 09:53:41,342] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:432) - Infer model path: C:\MLapp008\radiology\model\pretrained_localization_vertebra.pt [2024-10-30 09:53:41,885] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - INV - Run Transform(s) [2024-10-30 09:53:41,885] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - INV - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred'] [2024-10-30 09:53:42,233] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (CropForegroundd): Time: 0.347; image: torch.Size([25, 330, 330, 359])(torch.float32); label: torch.Size([1, 330, 330, 358])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32) [2024-10-30 09:54:23,117] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - INV - Transform (Spacingd): Time: 40.8844; image: torch.Size([25, 512, 512, 156])(torch.float32); label: torch.Size([1, 512, 512, 156])(torch.float32); pred: torch.Size([25, 94, 109, 234])(torch.float32) [2024-10-30 09:54:23,162] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2024-10-30 09:54:23,163] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model', 'image', 'result_extension', 'result_dtype', 'client_id', 'description', 'image_path', 'label', 'pipeline_mode', 'latencies', 'image_cached', 'foreground_start_coord', 'foreground_end_coord', 'pred'] [2024-10-30 09:54:23,164] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (EnsureTyped): Time: 0.001; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32) [2024-10-30 09:54:23,584] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Activationsd): Time: 0.42; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([25, 512, 512, 156])(torch.float32) [2024-10-30 09:54:25,283] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (AsDiscreted): Time: 1.698; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 09:54:32,057] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (KeepLargestConnectedComponentd): Time: 6.774; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([1, 512, 512, 156])(torch.float32) [2024-10-30 09:54:32,058] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (Restored): Time: 0.0; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 09:54:32,921] [26228] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (VertebraLocalizationSegmentation): Time: 0.862; image: torch.Size([1, 94, 109, 234])(torch.float32); label: torch.Size([1, 94, 109, 234])(torch.float32); pred: torch.Size([512, 512, 156])(torch.float32) [2024-10-30 09:54:32,922] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:335) - ++ Latencies => Total: 52.0994; Pre: 0.5160; Inferer: 0.5010; Invert: 41.3184; Post: 9.7610; Write: 0.0000 [2024-10-30 09:54:32,922] [26228] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:371) - Result Json Keys: ['centroids', 'label_names', 'latencies'] [2024-10-30 09:54:32,922] [26228] [MainThread] [INFO] (root:176) - locate_vertebra completed. Centroids: {} [2024-10-30 09:54:32,923] [26228] [MainThread] [INFO] (root:179) - Starting segment_vertebra stage [2024-10-30 09:54:32,923] [26228] [MainThread] [INFO] (root:93) - Starting vertebra segmentation for 0 centroids [2024-10-30 09:54:32,930] [26228] [MainThread] [INFO] (root:94) - Initial memory usage: 9.7% [2024-10-30 09:54:32,931] [26228] [MainThread] [ERROR] (root:151) - Error in vertebra segmentation: No valid segmentation produced [2024-10-30 09:54:32,937] [26228] [MainThread] [INFO] (root:155) - Final memory usage: 9.7%

SachidanandAlle commented 3 weeks ago

@diazandr3s can you please help