How to predict and evaluate on a self-made test set

[MOMOANNIE]

:question: Question

Hello, I have my own test set and need to predict and evaluate it. When I evaluate it, the following error occurs. How do I need to preprocess the labels?

[mibaumgartner]

The labels need to be placed into "raw_splitted/labelsTs" and the preprocessing needs to be rerun. The code in scripts/preprocess.py can also be modified to only run the preprocessing on the labels. A more convenient solution will be provided in our next release.

[MOMOANNIE]

The labels need to be placed into "raw_splitted/labelsTs" and the preprocessing needs to be rerun. The code in scripts/preprocess.py can also be modified to only run the preprocessing on the labels. A more convenient solution will be provided in our next release.

Ok, I have put the data and labels of the test set under “raw_splitted/”, how do I modify scripts/preprocess.py to preprocess the labels separately?

Do you process the label of the test by entering if cfg["data"]["test_labels"]: this condition? Or directly change test=False to test =True，as follows?

[mibaumgartner]

Here is a code snipped to preprocess the labels:

@env_guard
def main_prep_labels():
    """
    Prepare (non preprocessed) labels
    """
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "tasks",
        type=str,
        nargs="+",
        help="Single or multiple task identifiers to process consecutively",
    )
    parser.add_argument(
        "-np",
        "--num_processes",
        type=int,
        default=4,
        required=False,
        help="Number of processes to use for croppping.",
    )

    args = parser.parse_args()
    tasks = args.tasks
    num_processes = args.num_processes

    for task in tasks:
        task_path = get_task(task)
        create_labels(
            source_dir=task_path / "raw_splitted",
            preprocessed_output_dir=task_path / "preprocessed",
            num_processes=num_processes,
        )

[MOMOANNIE]

Here is a code snipped to preprocess the labels:

@env_guard
def main_prep_labels():
    """
    Prepare (non preprocessed) labels
    """
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "tasks",
        type=str,
        nargs="+",
        help="Single or multiple task identifiers to process consecutively",
    )
    parser.add_argument(
        "-np",
        "--num_processes",
        type=int,
        default=4,
        required=False,
        help="Number of processes to use for croppping.",
    )

    args = parser.parse_args()
    tasks = args.tasks
    num_processes = args.num_processes

    for task in tasks:
        task_path = get_task(task)
        create_labels(
            source_dir=task_path / "raw_splitted",
            preprocessed_output_dir=task_path / "preprocessed",
            num_processes=num_processes,
        )

Ok, thank you very much, I have successfully predicted my self-made test set, and the prediction results have many _boxes.pkl files under test_predictions/. I have read these files and found that the files contain 'pred_boxes', 'pred_scores' , 'pred_labels', but I don't know what the numbers in 'pred_boxes' mean? Is it coordinates? Can I use these files to visualize the detection results?

[mibaumgartner]

Hi,

you can find information about the box format here: https://github.com/MIC-DKFZ/nnDetection/issues/156 and visualisation here: https://github.com/MIC-DKFZ/nnDetection/issues/16

Best, Michael

[MOMOANNIE]

Hi,

you can find information about the box format here: #156 and visualisation here: #16

Best, Michael

Hi Michael， Thank you for your patience in answering my question. Okay, I understand, I have successfully visualized my detection results and generated the corresponding json file. I read issues 156, and I still want to confirm with you that my understanding is correct. "score": represents the object contained in the box Confidence probability, "box": the order from top to bottom is x_min, y_min, x_max, y_max, z_min, z_max, is my understanding correct?

[mibaumgartner]

Yes, it is correct :) x,y,z refer to the array loaded via SimpleITK.

[MOMOANNIE]

Yes, it is correct :) x,y,z refer to the array loaded via SimpleITK.

Ok, I would also like to ask about the detection results are now displayed through the box as the picture one, whether there is a seg display?as the picture two?

[mibaumgartner]

nnDetection is currently only used for bounding box detection and does not include Instance Segmentation (which would be second image).

[MOMOANNIE]

nnDetection is currently only used for bounding box detection and does not include Instance Segmentation (which would be second image). Hi Michael， Ok, next, I will analyze the detected results carefully, thank you for your patience in helping me solve the problem！

[MOMOANNIE]

Hi,

you can find information about the box format here: #156 and visualisation here: #16

Best, Michael

Hi Michael I have a few questions when analyzing the test results

1. Which script does nndet_boxes2nii run?
2. Does the --threshold specified by nndet_boxes2nii refer to the score?
3. When using the nndet_boxes2nii command, is the IoU of the detection box and the real box 0.1 by default?
4. If I want to calculate the false positive rate of the self-made test set when the score is 0.3, 0.4, 0.5, 0.6, which analysis file should I use to get it?

[mibaumgartner]

Hi,

1) the script executes scripts/utils.py/boxes2nii 2) The threshold specifies the score from which the boxes should be shown 3) boxes2nii does not use any IoU thresholds, all predictions which lie above the score threshold are shown (except predictions which might overlap, e.g. if one box fully encloses another one only the bigger one will be visible if it has an higher score 4) nnDetection currently does not include False Positive Rate for specific score thresholds. You could draw inspiration from the FROC implementation: https://github.com/MIC-DKFZ/nnDetection/blob/main/nndet/evaluator/detection/froc.py

Do not use the analysis file for any kind of quantitative evaluation! It uses a simplified algorithm for matching and is not tested.

[MOMOANNIE]

Hi,

1. the script executes scripts/utils.py/boxes2nii
2. The threshold specifies the score from which the boxes should be shown
3. boxes2nii does not use any IoU thresholds, all predictions which lie above the score threshold are shown (except predictions which might overlap, e.g. if one box fully encloses another one only the bigger one will be visible if it has an higher score
4. nnDetection currently does not include False Positive Rate for specific score thresholds. You could draw inspiration from the FROC implementation: https://github.com/MIC-DKFZ/nnDetection/blob/main/nndet/evaluator/detection/froc.py

Do not use the analysis file for any kind of quantitative evaluation! It uses a simplified algorithm for matching and is not tested. Hi Michael Humm, I just read the .pkl file under/test_predictions/ and found that this file contains items such as "pred_boxes"and "pred_scores". I can use nndet_boxes2nii --threshold to filter the detection boxes with scores greater than the threshold, right? If I need to get the number of TP, FP, FN, TN under different scores (score>0.3, 0.4, 0.5, 0.6), how do I calculate it?I have obtained this information from the.pkl file under /test_predictions/:

[mibaumgartner]

Dear @MOMOANNIE ,

you can check the scripts for the FROC and the Histogram computation in the evaluation folder of nnDetection for some initial clues. We are currently not providing utilities to compute FP, FP, FN separately (TN do not exist in detection).

Best, Michael

[MOMOANNIE]

Dear @MOMOANNIE ,

you can check the scripts for the FROC and the Histogram computation in the evaluation folder of nnDetection for some initial clues. We are currently not providing utilities to compute FP, FP, FN separately (TN do not exist in detection).

Best, Michael

ok, thank you very much, I will close the issue