search

M3DV / RibFrac-Challenge

MICCAI 2020 RibFrac Challenge: Rib Fracture Detection and Classification (3D Instance Segmentation)
https://ribfrac.grand-challenge.org/
Apache License 2.0
49 stars 10 forks source link

About empty data in val dataset #4

Closed menggerSherry closed 4 years ago

menggerSherry commented 4 years ago

I use the val data set for evaluation. However, there are some errors when running to certain data. Does the evaluation.py file have any problems with the handling the NII files without fractures?

kaimingkuang commented 4 years ago

Could you copy the error information?

menggerSherry commented 4 years ago

Traceback (most recent call last): File "evaluation.py", line 528, in eval_results = evaluate(args.gt_dir, args.pred_dir) File "evaluation.py", line 484, in evaluate gt_arr, pred_arr, cur_gt_info, cur_pred_info)) File "evaluation.py", line 267, in evaluate_single_prediction gt_info, pred_info) File "evaluation.py", line 173, in _compile_pred_metrics clf_conf_mat = _get_clf_confusion_matrix(gt_info, pred_metrics) File "evaluation.py", line 86, in _get_clf_confusion_matrix conf_mat.loc["FN", gt_info["label_code"][i]] += 1 File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 873, in getitem return self._getitem_tuple(key) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 1044, in _getitem_tuple return self._getitem_lowerdim(tup) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 810, in _getitem_lowerdim return getattr(section, self.name)[new_key] File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 879, in getitem return self._getitem_axis(maybe_callable, axis=axis) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 1110, in _getitem_axis return self._get_label(key, axis=axis) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 1059, in _get_label return self.obj.xs(label, axis=axis) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/generic.py", line 3482, in xs loc = self.index.get_loc(key) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2891, in get_loc raise KeyError(key) from err KeyError: 'Background'

I don't know exactly where the problem is. The format and requirements of the file are as described above.

kaimingkuang commented 4 years ago

Is this the latest version of this repo? This seems to be the case that gt_info["label_code"][i] == "Background" and there is no such column as "Background" in the confusion matrix. In the current version, we have:

        if len(pred_metrics) == 0\
                or (gt_info.label_code[i] != "Background"\
                and gt_info.label_id[i] not in list(pred_metrics.hit_label)):
            conf_mat.loc["FN", gt_info["label_code"][i]] += 1

Thus, this line of code shouldn't be run if gt_info["label_code"][i] == "Background. Is your prediction also empty?

kaimingkuang commented 4 years ago

I see. This is because your prediction for that particular volume is empty (predicts no fracture at all). This is a bug and will be fixed soon.

kaimingkuang commented 4 years ago

Traceback (most recent call last): File "evaluation.py", line 528, in eval_results = evaluate(args.gt_dir, args.pred_dir) File "evaluation.py", line 484, in evaluate gt_arr, pred_arr, cur_gt_info, cur_pred_info)) File "evaluation.py", line 267, in evaluate_single_prediction gt_info, pred_info) File "evaluation.py", line 173, in _compile_pred_metrics clf_conf_mat = _get_clf_confusion_matrix(gt_info, pred_metrics) File "evaluation.py", line 86, in _get_clf_confusion_matrix conf_mat.loc["FN", gt_info["label_code"][i]] += 1 File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 873, in getitem return self._getitem_tuple(key) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 1044, in _getitem_tuple return self._getitem_lowerdim(tup) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 810, in _getitem_lowerdim return getattr(section, self.name)[new_key] File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 879, in getitem return self._getitem_axis(maybe_callable, axis=axis) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 1110, in _getitem_axis return self._get_label(key, axis=axis) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexing.py", line 1059, in _get_label return self.obj.xs(label, axis=axis) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/generic.py", line 3482, in xs loc = self.index.get_loc(key) File "/root/miniconda/envs/tf1.15_tr1.4.0_mx1.5.0_cu10.0_py36/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2891, in get_loc raise KeyError(key) from err KeyError: 'Background'

I don't know exactly where the problem is. The format and requirements of the file are as described above.

Just pushed a new commit. This bug should be fixed when your prediction is empty.

menggerSherry commented 4 years ago

Great. The problem is solved

kaimingkuang commented 4 years ago

Great. The problem is solved

Cool. I will close this issue then.