MIC-DKFZ / nnUNet

Apache License 2.0
5.79k stars 1.74k forks source link

nnUNetv2_plan_and_preprocess killed by OS due to running out of RAM. #1869

Closed lemonwh closed 9 months ago

lemonwh commented 9 months ago

Plans were saved to /home/work/nnUNet/nnUNetFrame/DATASET/nnUNet_preprocessed/Dataset006_Lung/nnUNetPlans.json Preprocessing... Preprocessing dataset Dataset006_Lung Configuration: 2d... 0%| | 0/63 [00:13<?, ?it/s] Traceback (most recent call last): File "/home/work/anaconda3/envs/python3.9_torch1.10_cuda11.3/bin/nnUNetv2_plan_and_preprocess", line 8, in sys.exit(plan_and_preprocess_entry()) File "/home/work/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_entrypoints.py", line 197, in plan_and_preprocess_entry preprocess(args.d, args.overwrite_plans_name, args.c, np, args.verbose) File "/home/work/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 138, in preprocess preprocess_dataset(d, plans_identifier, configurations, num_processes, verbose) File "/home/work/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 116, in preprocess_dataset preprocessor.run(dataset_id, c, plans_identifier, num_processes=n) File "/home/work/nnUNet/nnunetv2/preprocessing/preprocessors/default_preprocessor.py", line 246, in run raise RuntimeError('Some background worker is 6 feet under. Yuck. \n' RuntimeError: Some background worker is 6 feet under. Yuck. OK jokes aside. One of your background processes is missing. This could be because of an error (look for an error message) or because it was killed by your OS due to running out of RAM. If you don't see an error message, out of RAM is likely the problem. In that case reducing the number of workers might help

constantinulrich commented 9 months ago

Did you try to reduce the number of workers for preprocessing? Change this as described here

QianLingjun commented 9 months ago

are you datasets is vary large ?