Segmentation Fault During Cortical Thickness Extraction on Brain MRI Images Using ANTsPynet

GayanSamuditha commented 2 months ago

@cookpa @ntustison

I am running cortical thickness extraction for Brain MRI images (a batch of 4000 MRI images) and encountered a segmentation fault during processing. The script ran successfully for several images but failed for many images: 07/ADNI_003_S_0908_MR_MPRGradWarpB1_CorrectionN3Scaled_Br_20081015153156378_S46187_I121108.nii.

The error message I received was: srun: error: compt374: task 0: Segmentation fault (core dumped)

    atropos = antspynet.deep_atropos(t1, do_preprocessing=True, verbose=True)

    kk_segmentation = atropos['segmentation_image']
    kk_segmentation[kk_segmentation == 4] = 3
    gray_matter = atropos['probability_images'][2]
    white_matter = atropos['probability_images'][3] + atropos['probability_images'][4]
    kk = ants.kelly_kapowski(s=kk_segmentation, g=gray_matter, w=white_matter, its=45, r=0.025, m=1.5, x=0, verbose=1)

    dkt = antspynet.desikan_killiany_tourville_labeling(t1, do_preprocessing=True, verbose=True)
    dkt_cortical_mask = ants.threshold_image(dkt, 1000, 3000, 1, 0)
    dkt = dkt_cortical_mask * dkt
    kk_mask = ants.threshold_image(kk, 0, 0, 0, 1)
    dkt_propagated = ants.iMath(kk_mask, "PropagateLabelsThroughMask", kk_mask * dkt)

    kk_regional_stats = ants.label_stats(kk, dkt_propagated)

Is this because of a problem in the MRI image or the computation problem? Or Allocated memory or CPU cores for the configurations?

ntustison commented 2 months ago

dkt labeling requires a lot of memory. Without knowing anything further, my guess is that you're running into memory problems there. You can help confirm this by actually verifying that that is where the seg fault is occuring.

GayanSamuditha commented 2 months ago

@ntustison thank you very much for the reply.

I have two questions.

Question 01: What kind of images do ANTsXnet and ANTsPy pipelines accept as input images? Means for example in ADNI any specific images only accepted? like - MPR; GradWarp; B1 Correction; Mask

Question 02: Is there any way to handle the memory problem?

I'm facing these errors for most of my images:

Do these errors mean my images do not apply the ANTs pre-processing Pipeline?

WARNING: In /project/itksource/Modules/Registration/Metricsv4/include/itkCorrelationImageToImageMetricv4HelperThreader.hxx, line 77 CorrelationImageToImageMetricv4HelperThreader (0x40bbe9c0): collected only zero points

WARNING: In /project/itksource/Modules/Numerics/Optimizersv4/include/itkObjectToObjectMetric.hxx, line 575 CorrelationImageToImageMetricv4 (0x2c4d1720): No valid points were found during metric evaluation. For image metrics, verify that the images overlap appropriately. For instance, you can align the image centers by translation. For point-set metrics, verify that the fixed points, once transformed into the virtual domain space, actually lie within the virtual domain.

.local/lib/python3.10/site-packages/antspynet/utilities/desikan_killiany_tourville_labeling.py:388: RuntimeWarning: invalid value encountered in true_divide

ntustison commented 2 months ago

I have no problem running ADNI data on my machines. Perhaps your system is inadequate for running the DKT pipeline. I can try running a problematic image on my end as a check if you post it.

GayanSamuditha commented 2 months ago

@ntustison

Yes, you are right! This is causing on DKT pipeline. How did you manage to run the pipeline for multiple images? What kind of configurations did you use?

dkt = antspynet.desikan_killiany_tourville_labeling(t1, do_preprocessing=True, verbose=True)

This is caused when I'm running 100 images subset from my original images set around 4000. I ran my code using shell script,

SBATCH --cpus-per-task=40

SBATCH --mem=240G

Time around - 100hrs
The problem is when I'm running the 100 images and 60-80 images running successfully and causing this problem on other images. Then I ran that problem causing images individually it's running fine.

I used a pool executor to run images one by one in the subset:

**# Use ProcessPoolExecutor for concurrency with ProcessPoolExecutor(max_workers=4) as executor: futures = {executor.submit(process_and_append_to_csv, t1_file, csv_file_path): t1_file for t1_file in image_files}

    for future in as_completed(futures):
        t1_file = futures[future]
        try:
            future.result()
            logging.info(f"Processed and appended KK regional stats for {t1_file} to CSV.")
        except Exception as e:
            logging.error(f"Failed to process {t1_file}: {e}")**

I want a suggestion on how can I run the pipeline for more images. Any concurrent processing way to handle the DKT labeling?

ntustison commented 2 months ago

I don't run it using a pool executor. I run it either in a for loop or send the image batch as individual jobs to the cluster.

GayanSamuditha commented 2 months ago

@ntustison Actually, I tried simple for loop and iterating through the images but it didn't extract all the images. Just only some of the images.

This is the log: 2024-09-19 12:35:53,352 - INFO - Processed and appended KK regional stats for /../../../../../ADNI_003_S_4152_MR_MT1__GradWarp__N3m_Br_20120629154010367_S142599_I313667.nii to CSV.srun: error: compt394: task 0: Segmentation fault (core dumped)

I thought this was due to the memory issue then I used a pool executor to handle one image at a time and that also caused the same problem. All images were not extracted. Just some of the images only.

This is the log:

2024-09-20 18:15:53,556 - ERROR - Failed to process /../.../ADNI1/SubSet03/ADNI_006_S_1130_MR_MPR-R__GradWarp__B1_Correction_Br_20080224141226574_S45203_I92234.nii: A process in the process pool was terminated abruptly while the future was running or pending.

In my log end its showing this:
2024-09-19 12:35:53,352 - INFO - Processed and appended KK regional stats for /../../../../../ADNI_003_S_4152_MR_MT1__GradWarp__N3m_Br_20120629154010367_S142599_I313667.nii to CSV.srun: error: compt394: task 0: Segmentation fault (core dumped)

/home/.../.local/lib/python3.10/site-packages/antspynet/utilities/desikan_killiany_tourville_labeling.py:318: RuntimeWarning:

invalid value encountered in true_divide

/home/.../.local/lib/python3.10/site-packages/antspynet/utilities/desikan_killiany_tourville_labeling.py:388: RuntimeWarning:

invalid value encountered in true_divide
2024-09-20 19:38:50,236 - INFO - Memory usage: 9857.36 MB

GayanSamuditha commented 2 months ago

@ntustison

This is what happened, it stopped suddenly running after some time. I'm not sure why, is because of a memory issue, If it is the issue, how can we run the pipeline for multiple images? Like more than 1000 images?

As you said, this is the way I used to iterate the images:

def main():
    initialize_csv_file(csv_file_path)  
    image_files = glob.glob(os.path.join(data_folder_path, "*.nii"))

    for t1_file in image_files:
        process_and_append_to_csv(t1_file, csv_file_path)
        [logging.info](http://logging.info/)(f"Processed and appended {t1_file} to CSV.")

if __name__ == "__main__":
    main()

ntustison commented 2 months ago

I've run the DKT pipeline on thousands of images on my university's cluster in the way I described i.e., on the cluster with submitting each subject processing as a single job.. In fact, here's a recent paper and here's the associated github repo for processing all ~50000 UKBB images.

I don't know for a fact that it's a memory issue but this appears to be a system-specific issue. I can only guess what the problem is and can't provide any further support than that.

GayanSamuditha commented 1 month ago

@ntustison Thank you so much for the clarification.

ANTsX / ANTsPyNet

Segmentation Fault During Cortical Thickness Extraction on Brain MRI Images Using ANTsPynet #137

SBATCH --cpus-per-task=40

SBATCH --mem=240G