Auto3DSeg: How to Set up Multiple Phase Data in Datalist JSON File?

[moonforsun]

Describe the bug I was able to run the Auto3DSeg based on the Task04_Hippocampus example. However, I was encountered the following bug on a customized dataset which I want to have multiple phase images as the network input. Any suggestion to how to use multiple phase data configured in the datalist json file?

Environment

================================ Printing MONAI config...

MONAI version: 1.0.1 Numpy version: 1.23.5 Pytorch version: 1.13.0+cu117 MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False MONAI rev id: 8271a193229fe4437026185e218d5b06f7c8ce69 MONAI file: /home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/init.py

Optional dependencies: Pytorch Ignite version: 0.4.10 Nibabel version: 3.2.0 scikit-image version: 0.19.3 Pillow version: 9.3.0 Tensorboard version: 2.11.0 gdown version: 4.5.3 TorchVision version: 0.14.0+cu117 tqdm version: 4.64.1 lmdb version: 1.3.0 psutil version: 5.9.4 pandas version: 1.5.1 einops version: 0.6.0 transformers version: 4.21.3 mlflow version: 2.0.1 pynrrd version: 1.0.0

For details about installing the optional dependencies, please visit: https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies

================================ Printing system config...

System: Linux Linux version: Ubuntu 18.04.6 LTS Platform: Linux-5.4.0-1069-aws-x86_64-with-glibc2.27 Processor: x86_64 Machine: x86_64 Python version: 3.8.2 Process name: python Command: ['/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/bin/python', '-c', 'import monai; monai.config.print_debug_info()'] Open files: [] Num physical CPUs: 24 Num logical CPUs: 48 Num usable CPUs: 48 CPU usage (%): [3.3, 3.0, 3.0, 2.6, 3.3, 3.0, 3.3, 3.0, 3.3, 2.6, 3.3, 3.0, 3.3, 3.0, 3.3, 3.3, 3.0, 3.0, 3.0, 3.0, 3.3, 3.0, 3.0, 3.0, 3.0, 3.3, 3.3, 3.0, 3.3, 3.0, 3.0, 3.0, 3.6, 3.0, 3.0, 3.3, 3.0, 3.0, 3.3, 3.0, 3.3, 3.0, 3.0, 3.3, 3.3, 3.3, 3.3, 97.7] CPU freq. (MHz): 1959 Load avg. in last 1, 5, 15 mins (%): [0.4, 4.9, 8.0] Disk usage (%): 38.7 Avg. sensor temp. (Celsius): UNKNOWN for given OS Total physical memory (GB): 186.7 Available memory (GB): 184.2 Used memory (GB): 1.0

================================ Printing GPU config...

Num GPUs: 4 Has CUDA: True CUDA version: 11.7 cuDNN enabled: True cuDNN version: 8500 Current device: 0 Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86'] GPU 0 Name: NVIDIA A10G GPU 0 Is integrated: False GPU 0 Is multi GPU board: False GPU 0 Multi processor count: 80 GPU 0 Total memory (GB): 22.2 GPU 0 CUDA capability (maj.min): 8.6 GPU 1 Name: NVIDIA A10G GPU 1 Is integrated: False GPU 1 Is multi GPU board: False GPU 1 Multi processor count: 80 GPU 1 Total memory (GB): 22.2 GPU 1 CUDA capability (maj.min): 8.6 GPU 2 Name: NVIDIA A10G GPU 2 Is integrated: False GPU 2 Is multi GPU board: False GPU 2 Multi processor count: 80 GPU 2 Total memory (GB): 22.2 GPU 2 CUDA capability (maj.min): 8.6 GPU 3 Name: NVIDIA A10G GPU 3 Is integrated: False GPU 3 Is multi GPU board: False GPU 3 Multi processor count: 80 GPU 3 Total memory (GB): 22.2 GPU 3 CUDA capability (maj.min): 8.6

Additional context Full trace

poetry run python -m monai.apps.auto3dseg AutoRunner run --input='./input.yaml' --work_dir='./LiverCrop' 2022-11-28 02:19:56,661 - INFO - ./LiverCrop does not exists. Creating... 2022-11-28 02:19:56,661 - INFO - ./LiverCrop created to save all results 2022-11-28 02:19:56,661 - INFO - Loading ./input.yaml for AutoRunner and making a copy in /home/ubuntu/Code/autoseg3d/LiverCrop/input.yaml 2022-11-28 02:19:56,663 - INFO - The output_dir is not specified. /home/ubuntu/Code/autoseg3d/LiverCrop/ensemble_output will be used to save ensemble predictions 2022-11-28 02:19:56,663 - INFO - Directory /home/ubuntu/Code/autoseg3d/LiverCrop/ensemble_output is created to save ensemble predictions 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 320/320 [06:45<00:00, 1.27s/it] algo_templates.tar.gz: 296kB [00:00, 712kB/s]
2022-11-28 02:26:47,886 - INFO - Downloaded: /tmp/tmpndcigyw9/algo_templates.tar.gz 2022-11-28 02:26:47,886 - INFO - Expected md5 is None, skip md5 check for file /tmp/tmpndcigyw9/algo_templates.tar.gz. 2022-11-28 02:26:47,886 - INFO - Writing into directory: /home/ubuntu/Code/autoseg3d/LiverCrop. 2022-11-28 02:26:55,159 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/segresnet2d_0 2022-11-28 02:27:02,219 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/segresnet2d_1 2022-11-28 02:27:09,540 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/dints_0 2022-11-28 02:27:16,466 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/dints_1 2022-11-28 02:27:23,697 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0 2022-11-28 02:27:30,749 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_1 2022-11-28 02:27:38,115 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/segresnet_0 2022-11-28 02:27:45,072 - INFO - /home/ubuntu/Code/autoseg3d/LiverCrop/segresnet_1 2022-11-28 02:27:45,075 - INFO - Launching: torchrun --nnodes=1 --nproc_per_node=4 /home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py run --config_file='/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/network.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_infer.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_train.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_validate.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/hyper_parameters.yaml' ['torchrun', '--nnodes=1', '--nproc_per_node=4', '/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py', 'run', "--config_file='/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/network.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_infer.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_train.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_validate.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/hyper_parameters.yaml'"] Traceback (most recent call last): File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 186, in _run_cmd normal_out = subprocess.run(cmd.split(), env=ps_environ, check=True, capture_output=True) File "/usr/local/lib/python3.8/subprocess.py", line 512, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['torchrun', '--nnodes=1', '--nproc_per_node=4', '/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py', 'run', "--config_file='/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/network.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_infer.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_train.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/transforms_validate.yaml','/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/configs/hyper_parameters.yaml'"]' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.8/runpy.py", line 193, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/lib/python3.8/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/apps/auto3dseg/main.py", line 22, in fire.Fire( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/apps/auto3dseg/auto_runner.py", line 586, in run self._train_algo_in_sequence(history) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/apps/auto3dseg/auto_runner.py", line 488, in _train_algo_in_sequence algo.train(self.train_params) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 203, in train return self._run_cmd(cmd, devices_info) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/monai/apps/auto3dseg/bundle_gen.py", line 191, in _run_cmd raise RuntimeError(f"subprocess call error {e.returncode}: {errors}, {output}") from e RuntimeError: subprocess call error 1: b'WARNING:torch.distributed.run:

---

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

---

Traceback (most recent call last): File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 409, in fire.Fire() File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, kwargs) File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 95, in run str_img = os.path.join(data_file_base_dir, list_train[_i]["image"]) File "/usr/local/lib/python3.8/posixpath.py", line 90, in join genericpath._check_arg_types(\'join\', a, p) File "/usr/local/lib/python3.8/genericpath.py", line 152, in _check_arg_types raise TypeError(f\'{funcname}() argument must be str, bytes, or \' TypeError: join() argument must be str, bytes, or os.PathLike object, not \'list\' Traceback (most recent call last): File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 409, in fire.Fire() File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire Traceback (most recent call last): File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 409, in component, remaining_args = _CallAndUpdateTrace( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace fire.Fire() File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component = fn(varargs, kwargs) File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 95, in run component_trace = _Fire(component, args, parsed_flag_args, context, name) str_img = os.path.join(data_file_base_dir, list_train[_i]["image"]) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire

File "/usr/local/lib/python3.8/posixpath.py", line 90, in join genericpath._check_arg_types(\'join\', a, p) File "/usr/local/lib/python3.8/genericpath.py", line 152, in _check_arg_types raise TypeError(f\'{funcname}() argument must be str, bytes, or \' TypeError: join() argument must be str, bytes, or os.PathLike object, not \'list\' component, remaining_args = _CallAndUpdateTrace( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(varargs, kwargs) File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 95, in run str_img = os.path.join(data_file_base_dir, list_train[_i]["image"]) File "/usr/local/lib/python3.8/posixpath.py", line 90, in join genericpath._check_arg_types(\'join\', a, p) File "/usr/local/lib/python3.8/genericpath.py", line 152, in _check_arg_types raise TypeError(f\'{funcname}() argument must be str, bytes, or \' TypeError: join() argument must be str, bytes, or os.PathLike object, not \'list\' Traceback (most recent call last): File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 409, in fire.Fire() File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace component = fn(varargs, kwargs) File "/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py", line 95, in run str_img = os.path.join(data_file_base_dir, list_train[_i]["image"]) File "/usr/local/lib/python3.8/posixpath.py", line 90, in join genericpath._check_arg_types(\'join\', a, p) File "/usr/local/lib/python3.8/genericpath.py", line 152, in _check_arg_types raise TypeError(f\'{funcname}() argument must be str, bytes, or \' TypeError: join() argument must be str, bytes, or os.PathLike object, not \'list\' ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 34440) of binary: /home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/bin/python Traceback (most recent call last): File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/bin/torchrun", line 8, in sys.exit(main()) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper return f(args, **kwargs) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main run(args) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run elastic_launch( File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/ubuntu/.cache/pypoetry/virtualenvs/qbdl-autoseg3d-a3-JfBFH-py3.8/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/home/ubuntu/Code/autoseg3d/LiverCrop/swinunetr_0/scripts/train.py FAILED

Failures: [1]: time : 2022-11-28_02:27:51 host : ip-172-20-253-208.us-west-2.compute.internal rank : 1 (local_rank: 1) exitcode : 1 (pid: 34441) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [2]: time : 2022-11-28_02:27:51 host : ip-172-20-253-208.us-west-2.compute.internal rank : 2 (local_rank: 2) exitcode : 1 (pid: 34442) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html [3]: time : 2022-11-28_02:27:51 host : ip-172-20-253-208.us-west-2.compute.internal rank : 3 (local_rank: 3) exitcode : 1 (pid: 34443) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure): [0]: time : 2022-11-28_02:27:51 host : ip-172-20-253-208.us-west-2.compute.internal rank : 0 (local_rank: 0) exitcode : 1 (pid: 34440) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

', b'[info] number of GPUs: 4 [info] number of GPUs: 4 [info] number of GPUs: 4 2022-11-28 02:27:50,522 - Added key: store_based_barrier_key:1 to store for rank: 2 [info] number of GPUs: 4 2022-11-28 02:27:50,564 - Added key: store_based_barrier_key:1 to store for rank: 1 2022-11-28 02:27:50,570 - Added key: store_based_barrier_key:1 to store for rank: 0 2022-11-28 02:27:50,572 - Added key: store_based_barrier_key:1 to store for rank: 3 2022-11-28 02:27:50,572 - Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. [info] world_size: 4 2022-11-28 02:27:50,574 - Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. [info] world_size: 4 2022-11-28 02:27:50,574 - Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. [info] world_size: 4 2022-11-28 02:27:50,581 - Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes. [info] world_size: 4 '

[myron]

Unfortunately, at this moment, it's not supported, the "image" key should a string (a single image file). But we should be improving it soon.

[dongyang0122]

hi @moonforsun, thank you for raising the issue! we do not currently have the support for the list of image filenames, but it is in our development plan. You can still try Auto3DSeg with multi-phase or multi-modality images. But some modification on data formats is required. As Auto3DSeg supports datasets of MSD. You can check how the multi-phase data is stored in Task01 and Task05. Normally the multi-phase images are saved as a 4D nifti file. The dimension of the 4D matrix is (dim_x, dim_y, dim_z, c). Once the data is converted, you can run Auto3DSeg without issues on data loading.

[moonforsun]

@dongyang0122 Hi Dong, the 4D nifti file is helpful and can be run successfully. Thank you very much!