Closed rajeshtims closed 1 year ago
SachidanandAlle
commented
1 year ago 2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:225) - Using Multi GPU: True; GPUS: [0]
Bundles support is limited.. specially for multi-gpu as it tries to invoke shell script... so, disable multi-gpu and run.. (Train Option should give you that checkbox to disable multi-gpu)
rajeshtims
commented
1 year ago It still failed even though I tried turning the multigpu check box off; I also tried clearing the gpus : all box.
Somehow in the training request, it says 'gpus':'all' and 'force_multi_gpu': True
[2023-05-25 21:58:35,016] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:210) - Train Request: {'model': 'hovernet_nuclei', 'multi_gpu': False, 'gpus': 'all', 'bundle_path': 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification', 'local_rank': 0, 'force_multi_gpu': True}
Here is the full error:
[2023-05-25 22:00:37,881] [26476] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:103) - Final Records with hovernet patches: 2009 [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:180) - Total Records in Dataset: 2009; Validation Split: 0.2 [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:190) - Total Records for Training: 1607 [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:191) - Total Records for Validation: 402 [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:225) - Using Multi GPU: True; GPUS: [0] [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:226) - CUDA_VISIBLE_DEVICES: 0 [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:229) - Using device: cuda; Type: <class 'str'> [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:240) - (Experiment Management) Tracking: mlflow [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:241) - (Experiment Management) Tracking URI: [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:242) - (Experiment Management) Experiment Name: hovernet_nuclei [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:243) - (Experiment Management) Run Name: None [2023-05-25 22:00:37,930] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:307) - Using CUDA_VISIBLE_DEVICES: 0 [2023-05-25 22:00:37,930] [26476] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:125) - +++++++++++ Running STAGE 0......................... [2023-05-25 22:00:37,931] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:357) - RUNNING COMMAND:: ['torchrun', '--standalone', '--nnodes=1', '--nproc_per_node=1', '-m', 'monai.bundle', 'run', 'run', '--meta_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\metadata.json', '--config_file', "['C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\monailabel_train.json','C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\multi_gpu_train.json']", '--logging_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\logging.conf', '--tracking', 'mlflow', '--stage', '0', '--network_def#freeze_encoder', 'true', '--network_def#pretrained_url', 'file:///C:/Users/raj/apps/pathology/model/pathology_nuclei_segmentation_classification/models/stage0/model.pt'] failed to create process. [2023-05-25 22:00:37,938] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:365) - Return code: 0 [2023-05-25 22:00:37,938] [26476] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:133) - +++++++++++ Running STAGE 1......................... [2023-05-25 22:00:37,938] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:357) - RUNNING COMMAND:: ['torchrun', '--standalone', '--nnodes=1', '--nproc_per_node=1', '-m', 'monai.bundle', 'run', 'run', '--meta_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\metadata.json', '--config_file', "['C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\monailabel_train.json','C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\multi_gpu_train.json']", '--logging_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\logging.conf', '--tracking', 'mlflow', '--stage', '1', '--network_def#freeze_encoder', 'false', '--network_def#pretrained_url', 'None'] failed to create process. [2023-05-25 22:00:37,943] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:365) - Return code: 0 [2023-05-25 22:00:37,943] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:339) - Training Finished.... [2023-05-25 22:00:37,943] [26476] [MainThread] [INFO] (main:61) - Result: {} [2023-05-25 22:00:38,539] [14612] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 0
SachidanandAlle
commented
1 year ago The logs says it's still switching to multi GPU..
alidation: 402 [2023-05-25 22:00:37,888] [26476] [MainThread] [INFO] (monailabel.tasks.train.bundle:225) - Using Multi GPU: True; GPUS: [0]
rajeshtims
commented
1 year ago Yes, but I turned the multi gpu option off. I am not sure why this is happening. I tried looking into the conf files; I don't see any force_multigpu option either
SachidanandAlle
commented
1 year ago Meanwhile you can try to run training for other simple models... Like nuclei or segmentation models..
rajeshtims
commented
1 year ago All of the other models run without any hiccups. Here is a screenshot of segmentation_nuclei model running with multi gpu option turned on. I am kinda puzzled what's going on with hovernet_nuclei model.
SachidanandAlle
commented
1 year ago I think I remember there is an issue with hovernet bundle to run on windows.. you can raise an issue on the model zoo on this.. probably people can get it fixed to run on windows..
you can give a try it to run using Ubuntu (WSL) inside windows as an APP.. I didn't try.. so can't confirm.. but technically it should work
rajeshtims
commented
1 year ago Thank you very much. I just raised an issue on model zoo.
rajeshtims
commented
1 year ago Just in case someone lands here, here is the workaround:
I got the following error while training hovernet_nuclei model:
I am using QuPath with MONAILabel extension to train hovernet_nuclei model. I am using Anaconda in windows 10 and the python is version 3.9.16.
I can get annotations fine, saving them is also working; patches are extracted when I initiate training, but as soon as the patches get extracted, I am getting "failed to create process" message. It returns code 0 and just says training finished
I am using monailabel 0.7.0rc7; I tried recreating the environment; tried with redownloading the sample app. No luck so far.
Here is the entire error message:
(monailabel) C:\Users\raj>monailabel start_server --app C:/Users/raj/apps/pathology --studies C:/Users/raj/datasets/ --conf models hovernet_nuclei --limit_concurrency 10 Using PYTHONPATH=C:\Users\raj\Anaconda3\envs;C:\Users\raj\Anaconda3\envs;C:\Users\raj\Anaconda3 "" [2023-05-25 13:06:54,299] [11928] [MainThread] [INFO] (main:285) - USING:: version = False [2023-05-25 13:06:54,299] [11928] [MainThread] [INFO] (main:285) - USING:: app = C:\Users\raj\apps\pathology [2023-05-25 13:06:54,300] [11928] [MainThread] [INFO] (main:285) - USING:: studies = C:\Users\raj\datasets [2023-05-25 13:06:54,302] [11928] [MainThread] [INFO] (main:285) - USING:: verbose = INFO [2023-05-25 13:06:54,302] [11928] [MainThread] [INFO] (main:285) - USING:: conf = [['models', 'hovernet_nuclei']] [2023-05-25 13:06:54,302] [11928] [MainThread] [INFO] (main:285) - USING:: host = 0.0.0.0 [2023-05-25 13:06:54,303] [11928] [MainThread] [INFO] (main:285) - USING:: port = 8000 [2023-05-25 13:06:54,303] [11928] [MainThread] [INFO] (main:285) - USING:: uvicorn_app = monailabel.app:app [2023-05-25 13:06:54,303] [11928] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile = None [2023-05-25 13:06:54,304] [11928] [MainThread] [INFO] (main:285) - USING:: ssl_certfile = None [2023-05-25 13:06:54,304] [11928] [MainThread] [INFO] (main:285) - USING:: ssl_keyfile_password = None [2023-05-25 13:06:54,304] [11928] [MainThread] [INFO] (main:285) - USING:: ssl_ca_certs = None [2023-05-25 13:06:54,305] [11928] [MainThread] [INFO] (main:285) - USING:: workers = None [2023-05-25 13:06:54,305] [11928] [MainThread] [INFO] (main:285) - USING:: limit_concurrency = 10 [2023-05-25 13:06:54,305] [11928] [MainThread] [INFO] (main:285) - USING:: access_log = False [2023-05-25 13:06:54,306] [11928] [MainThread] [INFO] (main:285) - USING:: root_path = / [2023-05-25 13:06:54,306] [11928] [MainThread] [INFO] (main:285) - USING:: log_level = info [2023-05-25 13:06:54,306] [11928] [MainThread] [INFO] (main:285) - USING:: log_config = None [2023-05-25 13:06:54,307] [11928] [MainThread] [INFO] (main:285) - USING:: dryrun = False [2023-05-25 13:06:54,307] [11928] [MainThread] [INFO] (main:285) - USING:: action = start_server [2023-05-25 13:06:54,307] [11928] [MainThread] [INFO] (main:296) - Allow Origins: ['*'] [2023-05-25 13:06:55,237] [11928] [MainThread] [INFO] (uvicorn.error:74) - Started server process [11928] [2023-05-25 13:06:55,238] [11928] [MainThread] [INFO] (uvicorn.error:48) - Waiting for application startup. [2023-05-25 13:06:55,239] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.app:37) - Initializing App from: C:\Users\raj\apps\pathology; studies: C:\Users\raj\datasets; conf: {'models': 'hovernet_nuclei'} [2023-05-25 13:06:55,318] [11928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'> [2023-05-25 13:06:55,342] [11928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.classification_nuclei.ClassificationNuclei'> [2023-05-25 13:06:55,345] [11928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.hovernet_nuclei.HovernetNuclei'> [2023-05-25 13:06:55,347] [11928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.nuclick.NuClick'> [2023-05-25 13:06:55,349] [11928] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_nuclei.SegmentationNuclei'> [2023-05-25 13:06:55,350] [11928] [MainThread] [INFO] (main:83) - +++ Adding Model: hovernet_nuclei => lib.configs.hovernet_nuclei.HovernetNuclei 2023-05-25 13:06:55,350 - INFO - --- input summary of monai.bundle.scripts.download --- 2023-05-25 13:06:55,351 - INFO - > name: 'pathology_nuclei_segmentation_classification' 2023-05-25 13:06:55,351 - INFO - > version: '0.1.8' 2023-05-25 13:06:55,352 - INFO - > bundle_dir: 'C:\Users\raj\apps\pathology\model' 2023-05-25 13:06:55,352 - INFO - > source: 'github' 2023-05-25 13:06:55,354 - INFO - > removeprefix: 'monai' 2023-05-25 13:06:55,354 - INFO - > progress: True 2023-05-25 13:06:55,356 - INFO - ---
pathology_nuclei_segmentation_classification_v0.1.8.zip: 267MB [00:04, 57.9MB/s] 2023-05-25 13:07:00,206 - INFO - Downloaded: C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification_v0.1.8.zip 2023-05-25 13:07:00,206 - INFO - Expected md5 is None, skip md5 check for file C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification_v0.1.8.zip. 2023-05-25 13:07:00,207 - INFO - Writing into directory: C:\Users\raj\apps\pathology\model. [2023-05-25 13:07:01,748] [11928] [MainThread] [INFO] (main:87) - +++ Using Models: ['hovernet_nuclei'] [2023-05-25 13:07:01,748] [11928] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: C:\Users\raj\datasets [2023-05-25 13:07:01,750] [11928] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: True; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml'] [2023-05-25 13:07:01,766] [11928] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0 [2023-05-25 13:07:01,766] [11928] [MainThread] [INFO] (monailabel.datastore.local:151) - Start observing external modifications on datastore (AUTO RELOAD) [2023-05-25 13:07:02,293] [11928] [MainThread] [INFO] (main:129) - +++ Adding Inferer:: hovernet_nuclei => <lib.infers.hovernet_nuclei.HovernetNuclei object at 0x000000003BFC4B50> [2023-05-25 13:07:02,300] [11928] [MainThread] [INFO] (main:153) - +++ Adding Trainer:: hovernet_nuclei => <lib.trainers.hovernet_nuclei.HovernetNuclei object at 0x000000003BFB8730> [2023-05-25 13:07:02,300] [11928] [MainThread] [INFO] (main:174) - Active Learning Strategies:: ['wsi_random'] [2023-05-25 13:07:02,301] [11928] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\raj.cache\monailabel\sessions [2023-05-25 13:07:02,301] [11928] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600 [2023-05-25 13:07:02,302] [11928] [MainThread] [INFO] (monailabel.interfaces.app:469) - App Init - completed [2023-05-25 13:07:02,302] [timeloop] [INFO] Starting Timeloop.. [2023-05-25 13:07:02,302] [11928] [MainThread] [INFO] (timeloop:60) - Starting Timeloop.. [2023-05-25 13:07:02,303] [timeloop] [INFO] Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000000003BF97AF0>
[2023-05-25 13:07:02,303] [11928] [MainThread] [INFO] (timeloop:42) - Registered job <function MONAILabelApp.on_init_complete..run_scheduler at 0x000000003BF97AF0>
[2023-05-25 13:07:02,305] [timeloop] [INFO] Timeloop now started. Jobs will run based on the interval set
[2023-05-25 13:07:02,305] [11928] [MainThread] [INFO] (timeloop:63) - Timeloop now started. Jobs will run based on the interval set
[2023-05-25 13:07:02,307] [11928] [MainThread] [INFO] (uvicorn.error:62) - Application startup complete.
[2023-05-25 13:07:02,308] [11928] [MainThread] [INFO] (uvicorn.error:217) - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
[2023-05-25 13:13:21,763] [11928] [MainThread] [INFO] (monailabel.endpoints.activelearning:44) - Active Learning Request: {'strategy': 'wsi_random', 'image': '', 'patch_size': [1024, 1024], 'image_size': [0, 0]}
[2023-05-25 13:13:21,764] [11928] [MainThread] [INFO] (monailabel.tasks.activelearning.random:47) - Random: Selected Image: JP2K-33003-1; Weight: 809
[2023-05-25 13:13:21,768] [11928] [MainThread] [INFO] (monailabel.endpoints.activelearning:60) - Next sample: {'id': 'JP2K-33003-1', 'weight': 809, 'path': 'C:\Users\raj\datasets\JP2K-33003-1.svs', 'ts': 1685033742, 'name': 'JP2K-33003-1.svs', 'strategy': {'wsi_random': {'ts': 1685045601, 'client_id': 'admin'}}}
[2023-05-25 13:14:22,399] [11928] [MainThread] [INFO] (monailabel.endpoints.wsi_infer:109) - WSI Infer Request: {'model': 'hovernet_nuclei', 'image': 'JP2K-33003-1', 'output': 'asap', 'level': 0, 'location': [2340, 5175], 'size': [119, 118], 'tile_size': [1024, 1024], 'min_poly_area': 30, 'foreground': [], 'background': [], 'max_workers': 1}
[2023-05-25 13:14:22,420] [11928] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:277) - Infer Request (final): {'device': 'cuda:0', 'model_filename': ['model.pt'], 'label_colors': {'Other': (255, 0, 0), 'Inflammatory': (255, 255, 0), 'Epithelial': (0, 0, 255), 'Spindle-Shaped': (0, 255, 0)}, 'model': 'hovernet_nuclei', 'image': 'C:\Users\raj\datasets\JP2K-33003-1.svs', 'output': 'asap', 'level': 0, 'location': (2340, 5175), 'size': (119, 118), 'tile_size': [1024, 1024], 'min_poly_area': 30, 'foreground': [], 'background': [], 'max_workers': 1, 'id': 0, 'logging': 'INFO', 'result_write_to_file': False, 'description': 'A simultaneous segmentation and classification of nuclei within multitissue histology images based on CoNSeP data', 'save_label': False}
monai.transforms.io.dictionary LoadImaged.init:image_only: Current default value of argument provided model_file: C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\models\model.pt
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00, 1.58s/it]
monai.transforms.io.dictionary SaveImaged.init:resample: Current default value of argument
image_only=Falsehas been deprecated since version 1.1. It will be changed toimage_only=Truein version 1.3. [2023-05-25 13:14:22,430] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - PRE - Run Transform(s) [2023-05-25 13:14:22,430] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - PRE - Input Keys: ['device', 'model_filename', 'label_colors', 'model', 'image', 'output', 'level', 'location', 'size', 'tile_size', 'min_poly_area', 'foreground', 'background', 'max_workers', 'id', 'logging', 'result_write_to_file', 'description', 'save_label', 'image_path'] [2023-05-25 13:14:22,564] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (LoadImagePatchd): Time: 0.1334; image: torch.Size([119, 118, 3])(torch.uint8) [2023-05-25 13:14:22,565] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (EnsureChannelFirstd): Time: 0.0; image: torch.Size([3, 119, 118])(torch.uint8) [2023-05-25 13:14:23,736] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (CastToTyped): Time: 1.1701; image: torch.Size([3, 119, 118])(torch.float32) [2023-05-25 13:14:23,737] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - PRE - Transform (ScaleIntensityRanged): Time: 0.001; image: torch.Size([3, 119, 118])(torch.float32) [2023-05-25 13:14:23,740] [11928] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:480) - Inferer:: cuda:0 => SlidingWindowHoVerNetInferer => {'roi_size': 256, 'sw_batch_size': 16, 'overlap': 0.359375, 'mode': constant, 'sigma_scale': 0.125, 'padding_mode': 'constant', 'cval': 0, 'sw_device': None, 'device': None, 'progress': True, 'cpu_thresh': None, 'buffer_steps': None, 'buffer_dim': -1, 'roi_weight_map': None, 'extra_input_padding': (46, 46, 46, 46)} [2023-05-25 13:14:23,740] [11928] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:418) - Infer model path: C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\models\model.pt [2023-05-25 13:14:23,741] [11928] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:426) - Usingresample=Truehas been deprecated since version 1.1. It will be changed toresample=Falsein version 1.3. [2023-05-25 13:14:25,780] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:76) - POST - Run Transform(s) [2023-05-25 13:14:25,780] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:77) - POST - Input Keys: ['device', 'model_filename', 'label_colors', 'model', 'image', 'output', 'level', 'location', 'size', 'tile_size', 'min_poly_area', 'foreground', 'background', 'max_workers', 'id', 'logging', 'result_write_to_file', 'description', 'save_label', 'image_path', 'image_meta_dict', 'latencies', 'pred'] [2023-05-25 13:14:25,781] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (FlattenSubKeysd): Time: 0.0; image: torch.Size([3, 119, 118])(torch.float32); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) [2023-05-25 13:14:25,811] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (HoVerNetInstanceMapPostProcessingd): Time: 0.0289; image: torch.Size([3, 119, 118])(torch.float32); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) [2023-05-25 13:14:25,824] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (HoVerNetNuclearTypePostProcessingd): Time: 0.011; image: torch.Size([3, 119, 118])(torch.float32); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) [2023-05-25 13:14:25,824] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (RenameKeyd): Time: 0.0; image: torch.Size([3, 119, 118])(torch.float32); pred: torch.Size([1, 119, 118])(torch.int64); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) [2023-05-25 13:14:25,825] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (SqueezeDimd): Time: 0.0; image: torch.Size([3, 119, 118])(torch.float32); pred: torch.Size([119, 118])(torch.int64); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) Any labeled images will be returned as a boolean array. Did you mean to use a boolean array? Only one label was provided toremove_small_objects. Did you mean to use a boolean array? [2023-05-25 13:14:25,828] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (PostFilterLabeld): Time: 0.002; image: torch.Size([3, 119, 118])(torch.float32); pred: (119, 118)(int64); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) [2023-05-25 13:14:25,830] [11928] [MainThread] [INFO] (monailabel.interfaces.utils.transform:122) - POST - Transform (FindContoursd): Time: 0.001; image: torch.Size([3, 119, 118])(torch.float32); pred: (119, 118)(int64); horizontal_vertical: torch.Size([2, 119, 118])(torch.float32); nucleus_prediction: torch.Size([2, 119, 118])(torch.float32); type_prediction: torch.Size([5, 119, 118])(torch.float32) [2023-05-25 13:14:25,831] [11928] [MainThread] [INFO] (monailabel.transform.writer:291) - +++ Output Type: asap [2023-05-25 13:14:25,832] [11928] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:332) - ++ Latencies => Total: 3.4133; Pre: 1.3175; Inferer: 2.0330; Invert: 0.0000; Post: 0.0607; Write: 0.0010 [2023-05-25 13:14:25,832] [11928] [MainThread] [INFO] (monailabel.tasks.infer.basic_infer:357) - Result Json Keys: ['annotation', 'label_names', 'latencies'] [2023-05-25 13:14:25,833] [11928] [MainThread] [INFO] (monailabel.interfaces.app:692) - JP2K-33003-1 => 0 => cuda:0 => 1 / 1; Latencies: {'pre': 1.32, 'infer': 2.03, 'invert': 0.0, 'post': 0.06, 'write': 0.0, 'total': 3.41, 'transform': {'pre': {'LoadImagePatchd': 0.1334, 'EnsureChannelFirstd': 0.0, 'CastToTyped': 1.1701, 'ScaleIntensityRanged': 0.001}, 'post': {'FlattenSubKeysd': 0.0, 'HoVerNetInstanceMapPostProcessingd': 0.0289, 'HoVerNetNuclearTypePostProcessingd': 0.011, 'RenameKeyd': 0.0, 'SqueezeDimd': 0.0, 'PostFilterLabeld': 0.002, 'FindContoursd': 0.001}}} [2023-05-25 13:14:25,834] [11928] [MainThread] [INFO] (monailabel.interfaces.app:721) - +++ Generating ASAP XML Annotation [2023-05-25 13:14:25,835] [11928] [MainThread] [INFO] (monailabel.utils.others.pathology:125) - Annotation keys: dict_keys(['location', 'size', 'elements', 'labels']) [2023-05-25 13:14:25,835] [11928] [MainThread] [INFO] (monailabel.utils.others.pathology:132) - Adding Contours for label: Inflammatory; color: #ffff00; color_map: {'Inflammatory': (255, 255, 0), 'Spindle-Shaped': (0, 255, 0)} [2023-05-25 13:14:25,836] [11928] [MainThread] [INFO] (monailabel.utils.others.pathology:132) - Adding Contours for label: Spindle-Shaped; color: #00ff00; color_map: {'Inflammatory': (255, 255, 0), 'Spindle-Shaped': (0, 255, 0)} [2023-05-25 13:14:25,837] [11928] [MainThread] [INFO] (monailabel.utils.others.pathology:154) - Total Annotations: 7 [2023-05-25 13:15:38,438] [11928] [MainThread] [INFO] (monailabel.endpoints.datastore:68) - Image: JP2K-33003-1-patch-2340_5175_119_118; File: <starlette.datastructures.UploadFile object at 0x00000000468810A0>; params: {} [2023-05-25 13:15:38,439] [11928] [MainThread] [INFO] (monailabel.datastore.local:439) - Adding Image: JP2K-33003-1-patch-2340_5175_119_118 => C:\Users\raj\AppData\Local\Temp\tmpixdrlydf.png [2023-05-25 13:15:38,477] [11928] [MainThread] [INFO] (monailabel.endpoints.datastore:101) - Saving Label for JP2K-33003-1-patch-2340_5175_119_118 for tag: final by admin [2023-05-25 13:15:38,478] [11928] [MainThread] [INFO] (monailabel.endpoints.datastore:112) - Save Label params: {} [2023-05-25 13:15:38,478] [11928] [MainThread] [INFO] (monailabel.datastore.local:486) - Saving Label for Image: JP2K-33003-1-patch-2340_5175_119_118; Tag: final; Info: {} [2023-05-25 13:15:38,479] [11928] [MainThread] [INFO] (monailabel.datastore.local:494) - Adding Label: JP2K-33003-1-patch-2340_5175_119_118 => final => C:\Users\raj\AppData\Local\Temp\tmpq57i05iu.xml [2023-05-25 13:15:38,483] [11928] [MainThread] [INFO] (monailabel.datastore.local:510) - Label Info: {'ts': 1685045738, 'name': 'JP2K-33003-1-patch-2340_5175_119_118.xml'} [2023-05-25 13:15:38,485] [11928] [MainThread] [INFO] (monailabel.interfaces.app:493) - New label saved for: JP2K-33003-1-patch-2340_5175_119_118 => JP2K-33003-1-patch-2340_5175_119_118 [2023-05-25 13:15:38,510] [11928] [Thread-2] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0 [2023-05-25 13:15:38,526] [11928] [Thread-2] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0 [2023-05-25 13:16:07,822] [11928] [MainThread] [INFO] (monailabel.utils.async_tasks.task:41) - Train request: {'model': 'hovernet_nuclei', 'run_id': 'train', 'tracking_experiment_name': 'hov_seg', 'tracking_uri': 'http://127.0.0.1', 'bundle_path': 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification'} [2023-05-25 13:16:07,823] [11928] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:49) - Before:: C:\Users\raj\Anaconda3\envs;C:\Users\raj\Anaconda3\envs;C:\Users\raj\Anaconda3;C:\Users\raj\apps\pathology [2023-05-25 13:16:07,824] [11928] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:53) - After:: C:\Users\raj\Anaconda3\envs;C:\Users\raj\Anaconda3\envs;C:\Users\raj\Anaconda3;C:\Users\raj\apps\pathology;C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification [2023-05-25 13:16:07,825] [11928] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:65) - COMMAND:: C:\Users\raj\Anaconda3\envs\monailabel\python.exe -m monailabel.interfaces.utils.app -m train -r {"model":"hovernet_nuclei","run_id":"train","tracking_experiment_name":"hov_seg","tracking_uri":"http://127.0.0.1","bundle_path":"C:\\Users\\raj\\apps\\pathology\\model\\pathology_nuclei_segmentation_classification","gpus":"all"} [2023-05-25 13:16:08,323] [17280] [MainThread] [INFO] (main:37) - Initializing App from: C:\Users\raj\apps\pathology; studies: C:\Users\raj\datasets; conf: {'models': 'hovernet_nuclei'} [2023-05-25 13:16:12,279] [17280] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for MONAILabelApp Found: <class 'main.MyApp'> [2023-05-25 13:16:12,288] [17280] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.classification_nuclei.ClassificationNuclei'> [2023-05-25 13:16:12,288] [17280] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.hovernet_nuclei.HovernetNuclei'> [2023-05-25 13:16:12,289] [17280] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.nuclick.NuClick'> [2023-05-25 13:16:12,290] [17280] [MainThread] [INFO] (monailabel.utils.others.class_utils:57) - Subclass for TaskConfig Found: <class 'lib.configs.segmentation_nuclei.SegmentationNuclei'> [2023-05-25 13:16:12,290] [17280] [MainThread] [INFO] (main:83) - +++ Adding Model: hovernet_nuclei => lib.configs.hovernet_nuclei.HovernetNuclei [2023-05-25 13:16:12,290] [17280] [MainThread] [INFO] (main:87) - +++ Using Models: ['hovernet_nuclei'] [2023-05-25 13:16:12,290] [17280] [MainThread] [INFO] (monailabel.interfaces.app:135) - Init Datastore for: C:\Users\raj\datasets [2023-05-25 13:16:12,290] [17280] [MainThread] [INFO] (monailabel.datastore.local:130) - Auto Reload: False; Extensions: ['.nii.gz', '.nii', '.nrrd', '.jpg', '.png', '.tif', '.svs', '.xml'] [2023-05-25 13:16:12,307] [17280] [MainThread] [INFO] (monailabel.datastore.local:577) - Invalidate count: 0 [2023-05-25 13:16:12,807] [17280] [MainThread] [INFO] (main:129) - +++ Adding Inferer:: hovernet_nuclei => <lib.infers.hovernet_nuclei.HovernetNuclei object at 0x000000003A0E4B80> [2023-05-25 13:16:12,808] [17280] [MainThread] [INFO] (main:153) - +++ Adding Trainer:: hovernet_nuclei => <lib.trainers.hovernet_nuclei.HovernetNuclei object at 0x000000003A0E4D00> [2023-05-25 13:16:12,808] [17280] [MainThread] [INFO] (main:174) - Active Learning Strategies:: ['wsi_random'] [2023-05-25 13:16:12,808] [17280] [MainThread] [INFO] (monailabel.utils.sessions:51) - Session Path: C:\Users\raj.cache\monailabel\sessions [2023-05-25 13:16:12,809] [17280] [MainThread] [INFO] (monailabel.utils.sessions:52) - Session Expiry (max): 3600 [2023-05-25 13:16:12,809] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:210) - Train Request: {'model': 'hovernet_nuclei', 'run_id': 'train', 'tracking_experiment_name': 'hov_seg', 'tracking_uri': 'http://127.0.0.1', 'bundle_path': 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification', 'gpus': 'all', 'local_rank': 0, 'force_multi_gpu': True} [2023-05-25 13:16:12,813] [17280] [MainThread] [INFO] (lib.utils:90) - Split data based on tile size: (1024, 1024); groups: None 0%| | 0/16 [00:00<?, ?it/s][2023-05-25 13:16:12,815] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:12,815] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-39588_34378_1380_1380.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-39588_34378_1380_1380.xml'} [2023-05-25 13:16:12,819] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 1727 [2023-05-25 13:16:12,819] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-39588_34378_1380_1380 => Groups: dict_keys(['other', 'inflammatory', 'epithelial']); Location: (349, 308); Size: 774 x 768 [2023-05-25 13:16:12,864] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (1380, 1380, 3); Total Patches to save: 4 [2023-05-25 13:16:13,289] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 8; c: 1; unique: (array([0, 1], dtype=uint8), array([1901426, 2974], dtype=int64)) [2023-05-25 13:16:13,310] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([1901247, 3153], dtype=int64)) [2023-05-25 13:16:13,332] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 35; c: 1; unique: (array([0, 1], dtype=uint8), array([1881580, 22820], dtype=int64)) [2023-05-25 13:16:13,332] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (1380, 1380); Total Patches to save: 4 6%|6 | 1/16 [00:00<00:08, 1.81it/s][2023-05-25 13:16:13,366] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,367] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-42406_35461_546_546.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-42406_35461_546_546.xml'} [2023-05-25 13:16:13,369] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 1191 [2023-05-25 13:16:13,369] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-42406_35461_546_546 => Groups: dict_keys(['other', 'inflammatory', 'epithelial']); Location: (3, 2); Size: 522 x 523 [2023-05-25 13:16:13,376] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (546, 546, 3); Total Patches to save: 1 [2023-05-25 13:16:13,446] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 6; c: 1; unique: (array([0, 1], dtype=uint8), array([295577, 2539], dtype=int64)) [2023-05-25 13:16:13,449] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([295359, 2757], dtype=int64)) [2023-05-25 13:16:13,452] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 15; c: 1; unique: (array([0, 1], dtype=uint8), array([282091, 16025], dtype=int64)) [2023-05-25 13:16:13,453] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (546, 546); Total Patches to save: 1 [2023-05-25 13:16:13,462] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,462] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-43363_36028_376_543.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-43363_36028_376_543.xml'} [2023-05-25 13:16:13,463] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 438 [2023-05-25 13:16:13,464] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-43363_36028_376_543 => Groups: dict_keys(['other', 'epithelial']); Location: (2, 5); Size: 371 x 501 [2023-05-25 13:16:13,468] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (543, 376, 3); Total Patches to save: 1 [2023-05-25 13:16:13,520] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 5; c: 1; unique: (array([0, 1], dtype=uint8), array([202321, 1847], dtype=int64)) [2023-05-25 13:16:13,523] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 7; c: 1; unique: (array([0, 1], dtype=uint8), array([198400, 5768], dtype=int64)) [2023-05-25 13:16:13,523] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (543, 376); Total Patches to save: 1 19%|#8 | 3/16 [00:00<00:02, 4.90it/s][2023-05-25 13:16:13,531] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,532] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-42457_36044_822_512.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-42457_36044_822_512.xml'} [2023-05-25 13:16:13,534] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 888 [2023-05-25 13:16:13,534] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-42457_36044_822_512 => Groups: dict_keys(['other', 'epithelial']); Location: (54, 7); Size: 759 x 505 [2023-05-25 13:16:13,543] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (512, 822, 3); Total Patches to save: 1 [2023-05-25 13:16:13,642] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 8; c: 1; unique: (array([0, 1], dtype=uint8), array([417898, 2966], dtype=int64)) [2023-05-25 13:16:13,647] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 14; c: 1; unique: (array([0, 1], dtype=uint8), array([407784, 13080], dtype=int64)) [2023-05-25 13:16:13,647] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (512, 822); Total Patches to save: 1 25%|##5 | 4/16 [00:00<00:02, 5.66it/s][2023-05-25 13:16:13,657] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,657] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-41780_36342_657_725.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-41780_36342_657_725.xml'} [2023-05-25 13:16:13,658] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 791 [2023-05-25 13:16:13,659] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-41780_36342_657_725 => Groups: dict_keys(['other', 'epithelial']); Location: (22, 8); Size: 633 x 715 [2023-05-25 13:16:13,670] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (725, 657, 3); Total Patches to save: 1 [2023-05-25 13:16:13,770] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 4; c: 1; unique: (array([0, 1], dtype=uint8), array([474725, 1600], dtype=int64)) [2023-05-25 13:16:13,774] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 12; c: 1; unique: (array([0, 1], dtype=uint8), array([463974, 12351], dtype=int64)) [2023-05-25 13:16:13,775] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (725, 657); Total Patches to save: 1 31%|###1 | 5/16 [00:00<00:01, 6.22it/s][2023-05-25 13:16:13,784] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,784] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-17954_22620_177_177.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-17954_22620_177_177.xml'} [2023-05-25 13:16:13,787] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 1318 [2023-05-25 13:16:13,787] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-17954_22620_177_177 => Groups: dict_keys(['other', 'epithelial']); Location: (9, 4); Size: 168 x 172 [2023-05-25 13:16:13,789] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (177, 177, 3); Total Patches to save: 1 [2023-05-25 13:16:13,818] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 32; c: 1; unique: (array([0, 1], dtype=uint8), array([20422, 10907], dtype=int64)) [2023-05-25 13:16:13,818] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 3; c: 1; unique: (array([0, 1], dtype=uint8), array([19500, 11829], dtype=int64)) [2023-05-25 13:16:13,819] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (177, 177); Total Patches to save: 1 [2023-05-25 13:16:13,830] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,831] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-18098_22466_144_144.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-18098_22466_144_144.xml'} [2023-05-25 13:16:13,832] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 320 [2023-05-25 13:16:13,832] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-18098_22466_144_144 => Groups: dict_keys(['other', 'epithelial']); Location: (3, 6); Size: 139 x 135 [2023-05-25 13:16:13,833] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (144, 144, 3); Total Patches to save: 1 [2023-05-25 13:16:13,862] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 9; c: 1; unique: (array([0, 1], dtype=uint8), array([18510, 2226], dtype=int64)) [2023-05-25 13:16:13,862] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([17769, 2967], dtype=int64)) [2023-05-25 13:16:13,862] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (144, 144); Total Patches to save: 1 [2023-05-25 13:16:13,871] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,871] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-17917_22463_181_152.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-17917_22463_181_152.xml'} [2023-05-25 13:16:13,873] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 1049 [2023-05-25 13:16:13,873] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-17917_22463_181_152 => Groups: dict_keys(['other', 'epithelial']); Location: (1, 3); Size: 178 x 146 [2023-05-25 13:16:13,874] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (152, 181, 3); Total Patches to save: 1 [2023-05-25 13:16:13,902] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 27; c: 1; unique: (array([0, 1], dtype=uint8), array([18778, 8734], dtype=int64)) [2023-05-25 13:16:13,902] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 2; c: 1; unique: (array([0, 1], dtype=uint8), array([17899, 9613], dtype=int64)) [2023-05-25 13:16:13,902] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (152, 181); Total Patches to save: 1 50%|##### | 8/16 [00:01<00:00, 10.97it/s][2023-05-25 13:16:13,911] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,911] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-17678_22673_182_182.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-17678_22673_182_182.xml'} [2023-05-25 13:16:13,913] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 651 [2023-05-25 13:16:13,913] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-17678_22673_182_182 => Groups: dict_keys(['other', 'epithelial']); Location: (1, 3); Size: 178 x 179 [2023-05-25 13:16:13,914] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (182, 182, 3); Total Patches to save: 1 [2023-05-25 13:16:13,942] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 14; c: 1; unique: (array([0, 1], dtype=uint8), array([28476, 4648], dtype=int64)) [2023-05-25 13:16:13,943] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 5; c: 1; unique: (array([0, 1], dtype=uint8), array([27260, 5864], dtype=int64)) [2023-05-25 13:16:13,943] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (182, 182); Total Patches to save: 1 [2023-05-25 13:16:13,951] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:13,951] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6V4V-PL22-patch-16909_22665_284_284.png', 'label': 'C:\Users\raj\datasets\labels\final\6V4V-PL22-patch-16909_22665_284_284.xml'} [2023-05-25 13:16:13,956] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 2628 [2023-05-25 13:16:13,957] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6V4V-PL22-patch-16909_22665_284_284 => Groups: dict_keys(['other', 'epithelial']); Location: (1, 2); Size: 283 x 277 [2023-05-25 13:16:13,958] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (284, 284, 3); Total Patches to save: 1 [2023-05-25 13:16:13,995] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 52; c: 1; unique: (array([0, 1], dtype=uint8), array([61814, 18842], dtype=int64)) [2023-05-25 13:16:13,996] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 13; c: 1; unique: (array([0, 1], dtype=uint8), array([54470, 26186], dtype=int64)) [2023-05-25 13:16:13,996] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (284, 284); Total Patches to save: 1 [2023-05-25 13:16:14,007] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:14,007] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6UZ7-PL22-patch-26615_32544_778_778.png', 'label': 'C:\Users\raj\datasets\labels\final\6UZ7-PL22-patch-26615_32544_778_778.xml'} [2023-05-25 13:16:14,015] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 4273 [2023-05-25 13:16:14,016] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6UZ7-PL22-patch-26615_32544_778_778 => Groups: dict_keys(['inflammatory', 'epithelial', 'spindle-shaped']); Location: (2, 2); Size: 776 x 773 [2023-05-25 13:16:14,031] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (778, 778, 3); Total Patches to save: 1 [2023-05-25 13:16:14,188] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 4; c: 1; unique: (array([0, 1], dtype=uint8), array([604348, 936], dtype=int64)) [2023-05-25 13:16:14,196] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 58; c: 1; unique: (array([0, 1], dtype=uint8), array([552758, 52526], dtype=int64)) [2023-05-25 13:16:14,202] [17280] [MainThread] [INFO] (lib.utils:593) - spindle-shaped => p: 29; c: 1; unique: (array([0, 1], dtype=uint8), array([541258, 64026], dtype=int64)) [2023-05-25 13:16:14,202] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (778, 778); Total Patches to save: 1 69%|######8 | 11/16 [00:01<00:00, 10.41it/s][2023-05-25 13:16:14,216] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:14,216] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6UZ7-PL22-patch-28591_32657_887_887.png', 'label': 'C:\Users\raj\datasets\labels\final\6UZ7-PL22-patch-28591_32657_887_887.xml'} [2023-05-25 13:16:14,334] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 5964 [2023-05-25 13:16:14,336] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6UZ7-PL22-patch-28591_32657_887_887 => Groups: dict_keys(['other', 'inflammatory', 'epithelial', 'spindle-shaped']); Location: (2, 2); Size: 885 x 885 [2023-05-25 13:16:14,355] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (887, 887, 3); Total Patches to save: 1 [2023-05-25 13:16:14,562] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 4; c: 1; unique: (array([0, 1], dtype=uint8), array([785904, 865], dtype=int64)) [2023-05-25 13:16:14,570] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 5; c: 1; unique: (array([0, 1], dtype=uint8), array([784619, 2150], dtype=int64)) [2023-05-25 13:16:14,580] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 90; c: 1; unique: (array([0, 1], dtype=uint8), array([712140, 74629], dtype=int64)) [2023-05-25 13:16:14,589] [17280] [MainThread] [INFO] (lib.utils:593) - spindle-shaped => p: 40; c: 1; unique: (array([0, 1], dtype=uint8), array([696642, 90127], dtype=int64)) [2023-05-25 13:16:14,590] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (887, 887); Total Patches to save: 1 [2023-05-25 13:16:14,606] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:14,606] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\6UZ7-PL22-patch-26941_32692_351_351.png', 'label': 'C:\Users\raj\datasets\labels\final\6UZ7-PL22-patch-26941_32692_351_351.xml'} [2023-05-25 13:16:14,608] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 909 [2023-05-25 13:16:14,608] [17280] [MainThread] [INFO] (lib.utils:562) - ID: 6UZ7-PL22-patch-26941_32692_351_351 => Groups: dict_keys(['inflammatory', 'epithelial', 'spindle-shaped']); Location: (8, 3); Size: 306 x 348 [2023-05-25 13:16:14,611] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (351, 351, 3); Total Patches to save: 1 [2023-05-25 13:16:14,654] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([123013, 188], dtype=int64)) [2023-05-25 13:16:14,656] [17280] [MainThread] [INFO] (lib.utils:593) - epithelial => p: 15; c: 1; unique: (array([0, 1], dtype=uint8), array([109535, 13666], dtype=int64)) [2023-05-25 13:16:14,657] [17280] [MainThread] [INFO] (lib.utils:593) - spindle-shaped => p: 5; c: 1; unique: (array([0, 1], dtype=uint8), array([108302, 14899], dtype=int64)) [2023-05-25 13:16:14,658] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (351, 351); Total Patches to save: 1 81%|########1 | 13/16 [00:01<00:00, 7.39it/s][2023-05-25 13:16:14,667] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:14,667] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\JP2K-33003-1-patch-2319_5116_152_133.png', 'label': 'C:\Users\raj\datasets\labels\final\JP2K-33003-1-patch-2319_5116_152_133.xml'} [2023-05-25 13:16:14,668] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 265 [2023-05-25 13:16:14,668] [17280] [MainThread] [INFO] (lib.utils:562) - ID: JP2K-33003-1-patch-2319_5116_152_133 => Groups: dict_keys(['other', 'inflammatory', 'spindle-shaped']); Location: (16, 10); Size: 124 x 117 [2023-05-25 13:16:14,669] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (133, 152, 3); Total Patches to save: 1 [2023-05-25 13:16:14,694] [17280] [MainThread] [INFO] (lib.utils:593) - other => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([20083, 133], dtype=int64)) [2023-05-25 13:16:14,695] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 10; c: 1; unique: (array([0, 1], dtype=uint8), array([18515, 1701], dtype=int64)) [2023-05-25 13:16:14,695] [17280] [MainThread] [INFO] (lib.utils:593) - spindle-shaped => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([18402, 1814], dtype=int64)) [2023-05-25 13:16:14,695] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (133, 152); Total Patches to save: 1 [2023-05-25 13:16:14,703] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:14,703] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\JP2K-33003-1-patch-2257_5110_206_206.png', 'label': 'C:\Users\raj\datasets\labels\final\JP2K-33003-1-patch-2257_5110_206_206.xml'} [2023-05-25 13:16:14,705] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 638 [2023-05-25 13:16:14,705] [17280] [MainThread] [INFO] (lib.utils:562) - ID: JP2K-33003-1-patch-2257_5110_206_206 => Groups: dict_keys(['inflammatory', 'spindle-shaped']); Location: (6, 12); Size: 195 x 168 [2023-05-25 13:16:14,706] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (206, 206, 3); Total Patches to save: 1 [2023-05-25 13:16:14,738] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 22; c: 1; unique: (array([0, 1], dtype=uint8), array([38980, 3456], dtype=int64)) [2023-05-25 13:16:14,739] [17280] [MainThread] [INFO] (lib.utils:593) - spindle-shaped => p: 5; c: 1; unique: (array([0, 1], dtype=uint8), array([38540, 3896], dtype=int64)) [2023-05-25 13:16:14,739] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (206, 206); Total Patches to save: 1 [2023-05-25 13:16:14,748] [17280] [MainThread] [INFO] (lib.utils:550) - ++ Using Groups: {} [2023-05-25 13:16:14,748] [17280] [MainThread] [INFO] (lib.utils:552) - Fetching Image/Label : {'image': 'C:\Users\raj\datasets\JP2K-33003-1-patch-2340_5175_119_118.png', 'label': 'C:\Users\raj\datasets\labels\final\JP2K-33003-1-patch-2340_5175_119_118.xml'} [2023-05-25 13:16:14,749] [17280] [MainThread] [INFO] (lib.utils:560) - Total Points: 134 [2023-05-25 13:16:14,749] [17280] [MainThread] [INFO] (lib.utils:562) - ID: JP2K-33003-1-patch-2340_5175_119_118 => Groups: dict_keys(['inflammatory', 'spindle-shaped']); Location: (12, 36); Size: 89 x 78 [2023-05-25 13:16:14,750] [17280] [MainThread] [INFO] (lib.utils:616) - Image => Input: (118, 119, 3); Total Patches to save: 1 [2023-05-25 13:16:14,778] [17280] [MainThread] [INFO] (lib.utils:593) - inflammatory => p: 5; c: 1; unique: (array([0, 1], dtype=uint8), array([13198, 844], dtype=int64)) [2023-05-25 13:16:14,779] [17280] [MainThread] [INFO] (lib.utils:593) - spindle-shaped => p: 1; c: 1; unique: (array([0, 1], dtype=uint8), array([13127, 915], dtype=int64)) [2023-05-25 13:16:14,779] [17280] [MainThread] [INFO] (lib.utils:616) - Label => Input: (118, 119); Total Patches to save: 1 100%|##########| 16/16 [00:01<00:00, 10.24it/s] 100%|##########| 16/16 [00:01<00:00, 8.11it/s] [2023-05-25 13:16:14,787] [17280] [MainThread] [INFO] (lib.utils:105) - +++ Total Records: 19 [2023-05-25 13:16:14,787] [17280] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:57) - Split data (len: 19) based on each nuclei 0%| | 0/19 [00:00<?, ?it/s] 5%|5 | 1/19 [00:02<00:50, 2.79s/it] 11%|# | 2/19 [00:05<00:47, 2.77s/it] 16%|#5 | 3/19 [00:08<00:44, 2.77s/it] 21%|##1 | 4/19 [00:11<00:42, 2.80s/it] 26%|##6 | 5/19 [00:13<00:39, 2.79s/it] 32%|###1 | 6/19 [00:16<00:36, 2.80s/it] 37%|###6 | 7/19 [00:19<00:33, 2.79s/it] 42%|####2 | 8/19 [00:22<00:30, 2.80s/it] 47%|####7 | 9/19 [00:25<00:27, 2.79s/it] 53%|#####2 | 10/19 [00:27<00:25, 2.80s/it] 58%|#####7 | 11/19 [00:30<00:22, 2.78s/it] 63%|######3 | 12/19 [00:33<00:19, 2.78s/it] 68%|######8 | 13/19 [00:36<00:16, 2.77s/it] 74%|#######3 | 14/19 [00:39<00:13, 2.78s/it] 79%|#######8 | 15/19 [00:41<00:11, 2.78s/it] 84%|########4 | 16/19 [00:44<00:08, 2.79s/it] 89%|########9 | 17/19 [00:47<00:05, 2.79s/it] 95%|#########4| 18/19 [00:50<00:02, 2.83s/it] 100%|##########| 19/19 [00:53<00:00, 2.82s/it] 100%|##########| 19/19 [00:53<00:00, 2.79s/it] [2023-05-25 13:17:07,874] [17280] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:103) - Final Records with hovernet patches: 931 [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:180) - Total Records in Dataset: 931; Validation Split: 0.2 [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:190) - Total Records for Training: 745 [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:191) - Total Records for Validation: 186 [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:225) - Using Multi GPU: True; GPUS: [0] [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:226) - CUDA_VISIBLE_DEVICES: None [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:229) - Using device: cuda; Type: <class 'str'> [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:240) - (Experiment Management) Tracking: mlflow [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:241) - (Experiment Management) Tracking URI: http://127.0.0.1[2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:242) - (Experiment Management) Experiment Name: hov_seg [2023-05-25 13:17:07,881] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:243) - (Experiment Management) Run Name: None [2023-05-25 13:17:07,905] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:307) - Using CUDA_VISIBLE_DEVICES: 0 [2023-05-25 13:17:07,905] [17280] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:125) - +++++++++++ Running STAGE 0......................... [2023-05-25 13:17:07,905] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:357) - RUNNING COMMAND:: ['torchrun', '--standalone', '--nnodes=1', '--nproc_per_node=1', '-m', 'monai.bundle', 'run', 'train', '--meta_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\metadata.json', '--config_file', "['C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\monailabel_train.json','C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\multi_gpu_train.json']", '--logging_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\logging.conf', '--tracking', 'mlflow', '--tracking_uri', 'http://127.0.0.1', '--stage', '0', '--network_def#freeze_encoder', 'true', '--network_def#pretrained_url', 'file:///C:/Users/raj/apps/pathology/model/pathology_nuclei_segmentation_classification/models/stage0/model.pt'] failed to create process. [2023-05-25 13:17:07,915] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:365) - Return code: 0 [2023-05-25 13:17:07,915] [17280] [MainThread] [INFO] (lib.trainers.hovernet_nuclei:133) - +++++++++++ Running STAGE 1......................... [2023-05-25 13:17:07,916] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:357) - RUNNING COMMAND:: ['torchrun', '--standalone', '--nnodes=1', '--nproc_per_node=1', '-m', 'monai.bundle', 'run', 'train', '--meta_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\metadata.json', '--config_file', "['C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\monailabel_train.json','C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\multi_gpu_train.json']", '--logging_file', 'C:\Users\raj\apps\pathology\model\pathology_nuclei_segmentation_classification\configs\logging.conf', '--tracking', 'mlflow', '--tracking_uri', 'http://127.0.0.1', '--stage', '1', '--network_def#freeze_encoder', 'false', '--network_def#pretrained_url', 'None'] failed to create process. [2023-05-25 13:17:07,921] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:365) - Return code: 0 [2023-05-25 13:17:07,922] [17280] [MainThread] [INFO] (monailabel.tasks.train.bundle:339) - Training Finished.... [2023-05-25 13:17:07,922] [17280] [MainThread] [INFO] (main:61) - Result: {} [2023-05-25 13:17:08,499] [11928] [ThreadPoolExecutor-2_0] [INFO] (monailabel.utils.async_tasks.utils:83) - Return code: 0Here is the monaiconfig debug info:
================================ Printing MONAI config...
MONAI version: 1.2.0rc7 Numpy version: 1.24.3 Pytorch version: 1.12.1 MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False MONAI rev id: e084885c6d498ab02df8107d23aa159f742a2cdd MONAI file: C:\Users\raj\Anaconda3\envs\monailabel\lib\site-packages\monai__init__.py
Optional dependencies: Pytorch Ignite version: 0.4.11 ITK version: 5.3.0 Nibabel version: 5.1.0 scikit-image version: 0.20.0 Pillow version: 9.4.0 Tensorboard version: 2.13.0 gdown version: 4.7.1 TorchVision version: 0.13.1 tqdm version: 4.65.0 lmdb version: 1.4.1 psutil version: 5.9.5 pandas version: 2.0.1 einops version: 0.6.1 transformers version: NOT INSTALLED or UNKNOWN VERSION. mlflow version: 2.3.2 pynrrd version: 1.0.0
For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
================================ Printing system config...
System: Windows Win32 version: ('10', '10.0.17763', 'SP0', 'Multiprocessor Free') Win32 edition: EnterpriseS Platform: Windows-10-10.0.17763-SP0 Processor: Intel64 Family 6 Model 85 Stepping 7, GenuineIntel Machine: AMD64 Python version: 3.9.16 Process name: python.exe Command: ['python', '-c', 'import monai; monai.config.print_debug_info()'] Open files: [popenfile(path='C:\Windows\System32\en-US\kernel32.dll.mui', fd=-1), popenfile(path='C:\Windows\System32\en-US\KernelBase.dll.mui', fd=-1), popenfile(path='C:\Windows\System32\en-US\tzres.dll.mui', fd=-1)] Num physical CPUs: 8 Num logical CPUs: 16 Num usable CPUs: 16 CPU usage (%): [9.1, 0.9, 1.8, 0.4, 3.1, 0.9, 7.1, 0.4, 3.1, 2.2, 1.3, 2.2, 0.9, 0.9, 0.4, 81.9] CPU freq. (MHz): 3592 Load avg. in last 1, 5, 15 mins (%): [0.0, 0.0, 0.0] Disk usage (%): 65.5 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 63.7 Available memory (GB): 49.1 Used memory (GB): 14.6
================================ Printing GPU config...
Num GPUs: 1 Has CUDA: True CUDA version: 11.3 cuDNN enabled: True cuDNN version: 8302 Current device: 0 Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37'] GPU 0 Name: NVIDIA GeForce RTX 2080 Ti GPU 0 Is integrated: False GPU 0 Is multi GPU board: False GPU 0 Multi processor count: 68 GPU 0 Total memory (GB): 11.0 GPU 0 CUDA capability (maj.min): 7.5