Closed rooskraaijveld closed 2 years ago
mibaumgartner
commented
2 years ago Dear @rooskraaijveld,
most likely the error occurs earlier but is only printed at the end because the preprocessing uses multiprocessing. This is usually caused by inconsistent label files, i.e. there are instances in the mask which are not in the json file (or the other way around).
Best, Michael
rooskraaijveld
commented
2 years ago Thank you Michael!
Indeed, I solved it by correcting some of my labels!
However, running the preprocessing on my data with 1 object seems to work, but images with 2 or more objects does not work. I have checked the data and the json files but there doesn't seem to be any errors in that. This is the error message I get when I do preprocessing with images with 2 or more objects :
2022-02-26 11:04:33.256 | WARNING | nndet.planning.properties.instance:instance_class_and_region_sizes:213 - Instance lost. Found {1: 0, 2: 0} in properties but [2] in seg of case0010.
......
multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/opt/nnDetection/nndet/planning/properties/instance.py", line 160, in analyze_instances_per_case props["all_ious"], props["class_ious"] = case_ious(props["boxes"], props) File "/opt/nnDetection/nndet/planning/properties/instance.py", line 287, in case_ious class_ious[_c] = compute_each_iou(boxes[cls_box_indices]) IndexError: boolean index did not match indexed array along dimension 0; dimension is 1 but corresponding boolean dimension is 2 """
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/bin/nndet_prep", line 33, in
__
Is there something going wrong or should I make a different case per object.
Eg Image 1 has 2 objects --> case1 = image 1 and object 1 case2 = image 1 and object 1
Thanks for answering my questions :)
Best, Roos
mibaumgartner
commented
2 years ago Dear @rooskraaijveld,
Glad I could help. Please don't split the same patient into multiple cases, this would lead to wrong training and results.
Could you please double-check that specific case? There might be two reasons why this occurs: 1) The segmentation file only contains one instance, i.e. there is only one object annotated (from the warning it looks like it has the id "2"). The corresponding json file contains two instances though, namely 1 and 2. If there is indeed only a single instance, please change the segmentation mask so the id of the object is 1 and remove instance 2 from the json file. If there are two instances please make sure that the segmentation really contains two objects with id's 1 and 2 (i.e. np.unique should give you [0, 1, 2]). 2) The first instance got lost during the cropping phase. You could double-check if there was any cropping happening in the logs of the preprocessing. If this was indeed the case, please check the positioning of the instance visually. Usually only zero regions are cropped and the instance should not be removed there. If everything looks fine there as well we would need to go deeper since this shouldn't happen. 3) Please make sure to delete the raw_cropped folder after your changes, just to save that the data is really up to date.
Best, Michael
rooskraaijveld
commented
2 years ago Dear Michael,
Thank you so much! It worked :)
I have two more small questions.
Thank you again!
Best, Roos
mibaumgartner
commented
2 years ago Dear @rooskraaijveld ,
Thanks for the update.
1) Overlapping instances are not yet supported in nnDetection since they are somewhat of a special case in 3D. One case should always represent one patient (with all of its corresponding instances).
2) Yes, checkpoints will be saved automatically. You can continue the training by re-running the training command and set -o train.mode=resume (-o is only needed if no other overwrites are specified)
Best, Michael
For some reason, nndet_prep will fun for my dataset until nndet.planning.properties.instance:analyze_instances_per_case. When the last case has been reached, it gives the following error:
2022-02-22 17:28:51.450 | INFO | nndet.planning.properties.instance:analyze_instances_per_case:153 - Processing instance properties of case case0038 multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar return list(itertools.starmap(args[0], args[1])) File "/opt/nnDetection/nndet/planning/properties/instance.py", line 158, in analyze_instances_per_case instance_class_and_region_sizes(case_id, iseg, props, all_classes) File "/opt/nnDetection/nndet/planning/properties/instance.py", line 218, in instance_class_and_region_sizes i_cls = instance_classes[instance_id] KeyError: 2 """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/conda/bin/nndet_prep", line 33, in
sys.exit(load_entry_point('nndet', 'console_scripts', 'nndet_prep')())
File "/opt/nnDetection/nndet/utils/check.py", line 58, in wrapper
return func(*args, *kwargs)
File "/opt/nnDetection/scripts/preprocess.py", line 406, in main
run(OmegaConf.to_container(cfg, resolve=True),
File "/opt/nnDetection/scripts/preprocess.py", line 325, in run
run_dataset_analysis(cropped_output_dir=Path(cfg["host"]["cropped_output_dir"]),
File "/opt/nnDetection/scripts/preprocess.py", line 129, in run_datasetanalysis
= analyzer.analyze_dataset(properties)
File "/opt/nnDetection/nndet/planning/analyzer.py", line 80, in analyze_dataset
props.update(property_fn(self))
File "/opt/nnDetection/nndet/planning/properties/instance.py", line 46, in analyze_instances
props_per_case = run_analyze_instances(analyzer, all_classes)
File "/opt/nnDetection/nndet/planning/properties/instance.py", line 77, in run_analyze_instances
props = p.starmap(analyze_instances_per_case, zip(
File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(args, **kwds))
File "/opt/conda/lib/python3.8/multiprocessing/pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "/opt/nnDetection/nndet/planning/properties/instance.py", line 158, in analyze_instances_per_case
instance_class_and_region_sizes(case_id, iseg, props, all_classes)
File "/opt/nnDetection/nndet/planning/properties/instance.py", line 218, in instance_class_and_region_sizes
i_cls = instance_classes[instance_id]
In total, there are 40 cases and it stops at the last one. It doesn't seem to be anything in my dataset as reshuffling my data does not make a difference. Is there anyone who could help?