search

microsoft / InnerEye-DeepLearning

Medical Imaging Deep Learning library to train and deploy 3D segmentation models on Azure Machine Learning
https://aka.ms/innereyeoss
MIT License
558 stars 142 forks source link

New directory is not created while using flag --no-train #844

Open Adapter668 opened 1 year ago

Adapter668 commented 1 year ago

Is there an existing issue for this?

Bug summary

While using flag --no-train there is an error that path does not exist. I tried also using local_weights_path, but still the same error.

Code for reproduction

CUDA_VISIBLE_DEVICES=5 python InnerEye/ML/runner.py --model=DynamicUnetBones --no-train --local_weights_path=outputs/2023-05-19T071816Z_DynamicUnetBones/checkpoints/last.ckpt

Actual outcome

Traceback (most recent call last): File "InnerEye/ML/runner.py", line 467, in main() File "InnerEye/ML/runner.py", line 461, in main run(project_root=fixed_paths.repository_root_directory(), File "InnerEye/ML/runner.py", line 457, in run return runner.run() File "InnerEye/ML/runner.py", line 220, in run self.run_in_situ(azure_run_info) File "InnerEye/ML/runner.py", line 412, in run_in_situ self.ml_runner.run() File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 378, in run self.run_inference_for_lightning_models(checkpoint_paths_for_testing) File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/ML/run_ml.py", line 469, in run_inference_for_lightning_models with change_working_directory(self.container.outputs_folder): File "/home/.conda/envs/InnerEye3/lib/python3.8/contextlib.py", line 113, in enter return next(self.gen) File "/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/InnerEye/Common/common_util.py", line 436, in change_working_directory os.chdir(new_path) FileNotFoundError: [Errno 2] No such file or directory: '/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/outputs/2023-05-22T073814Z_DynamicUnetBones'

Error messages

FileNotFoundError: [Errno 2] No such file or directory: '/home/medical-models-nnUnet/newer/medical-models/InnerEye-DeepLearning/outputs/2023-05-22T073814Z_DynamicUnetBones'

Expected outcome

It should run predictions on test data and save the result either on the local_weights_path directory or the new path (doesn't matter for me)

System info

OS: NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.5 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

INNEREYE COMMIT HASH: commit 2877002d50d3a34d80f647c18cb561025d9066cc (HEAD -> main, origin/main, origin/HEAD)

CONDA ENV: name: InnerEye3 channels:

ant0nsc commented 1 year ago

Thanks @adapter668 - this should be easily fixed, would you like to give it a try? All you'd need to do is add self.container.outputs_folder.mkdir(parents=True, exist_ok=True) in the run_inference_for_lightning_models, before it calls change_working_directory(self.container.outputs_folder)

Adapter668 commented 1 year ago

Thanks, I'll try your solution :) I used workaround and added os.makedirs in common_util.py for now, but I guess it'll be nice in the future if the --no-train flag works without changing code

ant0nsc commented 1 year ago

Hi @Adapter668 , I was more thinking that you could create a PR with that change to integrate it for later use

Adapter668 commented 1 year ago

Yes, I know, I just need to have some time to test if it works fine and I'll create PR