Issues Running Segmentation Model on Windows Platform with cuDNN Errors

[chocolatetoast-chiu]

Hello!

I've been using this toolkit for a long time, and it has been incredibly helpful for testing deep learning brain tumor segmentation models! Thanks for your hard work 🙏!

Recently, I changed the platform from Ubuntu 20.04 to Windows and set up all the requirements (running all commands in the Anaconda Prompt terminal). Here are the steps I've taken:

1. Verified Docker and GPU setup:

   docker run --rm --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

   

2. Checked WSL version:

   wsl -l -v

   

3. Successfully ran the BraTs Preprocessor in CPU mode and finished preprocessing the example data.

Problem Description

When I attempted to run the segmentation, I encountered issues related to cuDNN:

1. Using mic-dkfz:

   • Encountered CUDNN_STATUS_MAPPING_ERROR.

2. Using scan-20:

   • Encountered an error stating that a valid cuDNN algorithm could not be found.

To verify the cuDNN installation in the container, I ran the container in interactive mode and printed the package information:

docker run --rm -it --gpus device=0 -v E:\creomed_Leon_BraTS-Toolkit\codes\BraTS-Toolkit\brats_toolkit:/app/data/ brats/scan-20 bash

python3 -c "import torch; print('PyTorch version:', torch.__version__); print('cuDNN version:', torch.backends.cudnn.version()); print('CUDA version:', torch.version.cuda)"

Output:

Running nvidia-smi inside the interactive terminal:

Request for Assistance

I am seeking guidance on whether the segmentation steps are correct for running under the Windows platform. The same process worked fine under Ubuntu, so I suspect there may be different considerations for running the segmentation model with GPU on Windows.

Any insights or suggestions for running the segmentation model with GPU support on Windows would be greatly appreciated.

Thank you!

P.S. Here is my script to run Segmentation

import os
import sys
import time
import datetime
from brats_toolkit.segmentor import Segmentor

# log
starttime = str(datetime.datetime.now().time())
print("*** starting at", starttime, "***")

# specify the folder path
selected_name_path = 'example'
parent_path = "E:\\creomed_Leon_BraTS-Toolkit\\data_BraTs\\"
folder_path = parent_path + "output_preprocessor_single\\" + selected_name_path

# algorithms we used to select for segmentation
cids = ["scan-20"]

for filename in os.listdir(folder_path):
    examName = filename
    print('********** Start processing ' + examName + ' **********')

    # input files
    if os.path.isdir(folder_path + "\\" + examName + "\\robex_brats-space\\"):
        t1File = folder_path + "\\" + examName + "\\robex_brats-space\\" + examName + "_robex_brats_t1.nii.gz"
        t1cFile = folder_path + "\\" + examName + "\\robex_brats-space\\" + examName + "_robex_brats_t1c.nii.gz"
        t2File = folder_path + "\\" + examName + "\\robex_brats-space\\" + examName + "_robex_brats_t2.nii.gz"
        flaFile = folder_path + "\\" + examName + "\\robex_brats-space\\" + examName + "_robex_brats_fla.nii.gz"
    elif os.path.isdir(folder_path + "\\" + examName + "\\hdbet_brats-space\\"):
        t1File = folder_path + "\\" + examName + "\\hdbet_brats-space\\" + examName + "_hdbet_brats_t1.nii.gz"
        t1cFile = folder_path + "\\" + examName + "\\hdbet_brats-space\\" + examName + "_hdbet_brats_t1c.nii.gz"
        t2File = folder_path + "\\" + examName + "\\hdbet_brats-space\\" + examName + "_hdbet_brats_t2.nii.gz"
        flaFile = folder_path + "\\" + examName + "\\hdbet_brats-space\\" + examName + "_hdbet_brats_fla.nii.gz"
    else:
        print('********** ERROR: ' + examName + ' doesn\'t be preprocessed successfully **********')
        sys.exit()

    # output
    outputFolder = parent_path + "\\output_segmentor\\" + selected_name_path + "\\" + examName + "\\"

    # execute it
    for cid in cids:
        # instantiate
        seg = Segmentor(verbose=True)
        try:
            outputFile = outputFolder + cid + ".nii.gz"
            seg.segment(
                t1=t1File,
                t2=t2File,
                t1c=t1cFile,
                fla=flaFile,
                cid=cid,
                outputPath=outputFile,
            )

        except Exception as e:
            print("error:", str(e))
            print("error occured for:", cid)

        seg = []

    time.sleep(3)
    print('********** Finished processing ' + examName + ' **********')

[chocolatetoast-chiu]

[Update]

I also tried to run the segmentation model under wsl2 Ubuntu terminal, the preprocessing using cpu can be run successfully. And the cuDNN algorithm still can't be captured with the scan-20 model.

I thought would it be possible for this GPU to be too new to run the inference under the docker container environment? Referring to this packages' versions:

[neuronflow]

Sorry for the late response.

Yes, the A6000 you are using seems too new or, in other words, does not have enough backward compatibility to support running these old algorithms.

You can beta-test our new package with 2023+ algorithms that should be supported by your GPU:

https://github.com/BrainLesion/BraTS