444 Fix multi-node issue

[yiheng-wang-nv]

Fixes #444 .

Description

This PR is used to fix the multi-gpu configs issue if running with multi-node. It also has a bit enhancement on the endoscopic_inbody_classification to add close json file step. This bundle's unit test is also added within the PR.

Status

Ready

Please ensure all the checkboxes:

• [x] Codeformat tests passed locally by running ./runtests.sh --codeformat.
• [ ] In-line docstrings updated.
• [ ] Update version and changelog in metadata.json if changing an existing bundle.
• [ ] Please ensure the naming rules in config files meet our requirements (please refer to: CONTRIBUTING.md).
• [ ] Ensure versions of packages such as monai, pytorch and numpy are correct in metadata.json.
• [ ] Descriptions should be consistent with the content, such as eval_metrics of the provided weights and TorchScript modules.
• [ ] Files larger than 25MB are excluded and replaced by providing download links in large_file.yml.
• [ ] Avoid using path that contains personal information within config files (such as use /home/your_name/ for "bundle_root").

[yiheng-wang-nv]

I'm working on using this branch to test with a multi-node machine

[Nic-Ma]

Hi @yiheng-wang-nv ,

Can you try?

"device": "$torch.device('cuda:' + os.environ['LOCAL_RANK'])"

I think it should work.

Thanks.

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

Hi @Nic-Ma , I tried all bundles with changed config files, and also run multi-node training experiments on a part of bundles:

1. spleen_deepedit_annotation
2. spleen_ct_segmentation
3. brats_mri_axial_slices_generative_diffusion
4. swin_unetr_btcv_segmentation

I think this PR is now ready, and could you please help to review it, thank!

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build

[yiheng-wang-nv]

/build