search

Deep-MI / FastSurfer

PyTorch implementation of FastSurferCNN
Apache License 2.0
435 stars 115 forks source link

Unmatched ROIs in predicted segmentation and provided FastSurfer_ColorLUT.tsv #514

Open gmeng92 opened 2 months ago

gmeng92 commented 2 months ago

Question/Support Request

I run the FastSurferCNN (--seg-only) to get the DKT atlas segementaion given t1W MR images.

By using my local images, I ended up get 88 unique ROI labels(including the background = 0), which is inconsistent with the ROI list as /coifg/FastSurfer_ColorLUT.tsv. Then I followed your tutorial to run the --seg-only code using the image you provided as in (https://github.com/Deep-MI/FastSurfer/blob/dev/Tutorial/Tutorial_FastSurferCNN_QuickSeg.ipynb), it turns out to 88 unique ROI labels again.

So the provided labels (in the tsp) seems unmatched with the predicted labels (from segmented images). Beisides, I noticed that in the tutorial notebook, I find the cell under "Click3" saying that 95 predicted labels are expected. And all those mismatches happen to be on the right hemisphere cortex. Can you help me elaborate those unmatched labels? Thanks

Screenshots / Log files

Screenshot 2024-04-19 at 1 57 17 PM

dkuegler commented 2 months ago

That is expected and correct behavior. The classes in config/FastSurfer_ColorLUT.tsv are the labels that are predicted by the neural network, but we do a Postprocessing step after. For example, some labels get laterilized there (which is why you have mismatched labels on one side).

Moreover, you will see that the indices of the segmentation classes are also very different between the FastSurfer_ColorLUT.tsv and the output image.

gmeng92 commented 2 months ago

Thanks for the quick reply! So in practice, we will still use the file config/FastSurfer_ColorLUT.tsv for training/inferencing through FastSurferCNN, right? And what happens in the Postprocessing step after?

I noticed only the right hemisphere is influenced and many important regions of interest (like rh-postcentral etc.) are missing in the segmentation. I further compared the performance between FastSurferCNN and FreeSurfer on 11 T1 images. The correlation indicates fastssurfer can do a better job on lh-ctx segmentation compared to the rh-ctx. That makes me hesitate to use the segmented results as qualified "features" for following analysis.

Screenshot 2024-04-19 at 3 20 50 PM Screenshot 2024-04-19 at 3 21 25 PM
dkuegler commented 2 months ago

If you are just using FastSurfer via run_fastsurfer.sh or run_prediction.py, you will not get results as defined by config/config/FastSurfer_ColorLUT.tsv. Only Inference.eval (and Inference.run) has the segmentations as defined by config/FastSurfer_ColorCUT.tsv ... this is mapped to the label space used by FreeSurfer in RunModelOnData.get_prediction:

https://github.com/Deep-MI/FastSurfer/blob/16335a9db7be4bb5190be3f57de252266ef3f2e0/FastSurferCNN/run_prediction.py#L398-L413

I am not really sure what you are really comparing right now; FreeSurfer segmentation with FastSurfer's aparc+DKTatlas.aseg.deep.mgz ?

It seems to me you are manually calling just Inference.run and skipping the post processing steps in RunModelOnData.get_prediction.

dkuegler commented 2 months ago

Also, I am not sure about which metrics you are using in your plots? Is this correlation of volumes?

Those numbers look awfully low... And with comparisons between FreeSurfer and FastSurfer it will always be difficult to say what is better if you compare them with respect to each other. Furthermore, cortical regions evaluated on the volume can always be very misleading.

All that said, we run per-release continuous validation of FastSurfer and make sure the metrics presented in both the FastSurfer and the FastSurferVINN paper are still satisfied. Specifically, this means Test-Retest metrics such as https://www.sciencedirect.com/science/article/pii/S1053811920304985#fig9 and Dice evaluations.

m-reuter commented 2 months ago

Yes, if you want to compare volumes for the structures, use the values in aseg.stats or aparc+aseg.stats after a full run of FastSurfer (including the surface module), as we correct some volumes with the surfaces similar to FreeSurfer.

gmeng92 commented 2 months ago

If you are just using FastSurfer via run_fastsurfer.sh or run_prediction.py, you will not get results as defined by config/config/FastSurfer_ColorLUT.tsv. Only Inference.eval (and Inference.run) has the segmentations as defined by config/FastSurfer_ColorCUT.tsv ... this is mapped to the label space used by FreeSurfer in RunModelOnData.get_prediction:

https://github.com/Deep-MI/FastSurfer/blob/16335a9db7be4bb5190be3f57de252266ef3f2e0/FastSurferCNN/run_prediction.py#L398-L413

I am not really sure what you are really comparing right now; FreeSurfer segmentation with FastSurfer's aparc+DKTatlas.aseg.deep.mgz ?

It seems to me you are manually calling just Inference.run and skipping the post processing steps in RunModelOnData.get_prediction.

Yes. Exactly. The results I showed are after running therun_prediction.py. only, and the comparison I performed is between the FreeSurfer output: aparc.DKTatlas+aseg.mgz and the FastSurfer output: aparc+DKTatlas.aseg.deep.mgz. I used the correlation to compare the consistence for each ROI. So I am pretty sure the problem is due to the skipping of the Postprocessing step you mentioned. And as m-reuter commented(https://github.com/Deep-MI/FastSurfer/issues/514#issuecomment-2067769322), I should make a full run of FastSurfer including the surface module. Thanks for both of your detailed explanations!

dkuegler commented 2 months ago

Even run_prediction.py should output valid segmentation files, so I guess I still don't quite understand what you mean, but I would recommend using run_fastsurfer.sh --seg_only if you wanted to run FastSurfer without the surface pipeline anyway...

gmeng92 commented 2 months ago

Well. I expected two things : the segmented image and the volume stats corresponding to the segmentation. In the earlier post, all the results were collected after I run the run_prediction.py. And the volumes stats is calculated based on the aparc+DKTatlas.aseg.deep.mgz. As you and m-reuter mentioned, you have a separated Post-processing step for the volumes estimation. I should use the values in aseg.stats or aparc+aseg.stats. The volumes stats should then comparable to the FreeSurfer estimated. Am I understanding it right?

dkuegler commented 2 months ago

That all sounds correct, the only thing that does not match up is that the statistics you presented up top make no sense as an output to run_prediction.py. The postprocessing is built into run_prediction.py. The error that you are presenting makes no sense to me. You should also use FreeSurferColorLUT.txt with the aparc.DKTatlas+aseg.deep.mgz to identify different labels (which is the same as FreeSurfer).

To achieve this, we recommend running fastsurfer with the run_fastsurfer.sh (--seg_only or not) to get the aseg.mgzor aparc.DKTatlas+aseg.*****.mgz. They should all have all labels (left and right) and should be complete. We NEVER save the data before applying the postprocessing steps in FastSurfer. This way your reported

And all those mismatches happen to be on the right hemisphere cortex.

Makes no sense to me, I have never seen this issue and I cannot reproduce this. I guess if it works fine for you now, we can close this issue, but if you have a file that has missing labels, please give us a short script that reproduces this. This should never happen.

gmeng92 commented 2 months ago

The example I was using is from the Tutorial Notebook (https://github.com/Deep-MI/FastSurfer/blob/dev/Tutorial/Complete_FastSurfer_Tutorial.ipynb). !curl -k https://surfer.nmr.mgh.harvard.edu/pub/data/tutorial_data/buckner_data/tutorial_subjs/140/mri/orig.mgz -o "{SETUP_DIR}140_orig.mgz" . I run both the FastSurfer and FreeSurfer to compare the segmented images between aparc.DKTatlas+aseg.deep.mgz (FastSurfer) and aparc.DKTatlas+aseg.mgz (FreeSurfer). Here are the scripts I ran to generate the resulted images:

  1. FastSurfer:
    • run_prediction.py : input_dir=/Users/Projects/naip_data/Example_image output_dir=/Users/Projects/naip_data/Example_image/FastSurfer_result_run_fastsurfer python3 run_prediction.py --t1 $input_dir/subject_140/example140_orig.mgz \ --asegdkt_segfile $output_dir/subject_140/aparc.DKTatlas+aseg.deep.mgz \ --conformed_name $output_dir/subject_140/subconformed.mgz \ --brainmask_name $output_dir/subject_140/mri/mask.mgz \ --aseg_name $output_dir/subject_140/mri/aseg.auto_noCCseg.mgz \ --sid subject_140 \ --seg_log $output_dir/subject_140/scripts/deep-seg.log \ --vox_size min \ --batch_size 1
    • N4_bias_correct.py python3 N4_bias_correct.py --in $output_dir/subject_140/subconformed.mgz \ --out $output_dir/subject_140/mri/corrected.mgz \ --rescale $output_dir/subject_140/mri/rescaled.mgz \ --aseg $output_dir/subject_140/aparc.DKTatlas+aseg.deep.mgz \ --threads 4
    • segstats.py: fastsurfercnndir=/Users/Projects/FastSurfer/FastSurferCNN output_dir=/Users/Projects/naip_data/Example_image/FastSurfer_result_run_fastsurfer python3 segstats.py --segfile $output_dir/subject_140/aparc.DKTatlas+aseg.deep.mgz \ --segstatsfile $output_dir/subject_140/segs.stats \ --normfile $output_dir/subject_140/mri/corrected.mgz \ --ids 2 4 5 7 8 10 11 12 13 14 15 16 17 18 24 26 28 31 41 43 44 46 47 49 50 51 52 53 54 58 60 63 77 251 252 253 254 255 1002 1003 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 1020 1021 1022 1023 1024 1025 1026 1027 1028 1029 1030 1031 1034 1035 2002 2003 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 2031 2034 2035 \ --lut $fastsurfercnndir/config/FreeSurferColorLUT.txt \ --empty \ --threads 4
  2. FreeSurfer:

All the codes are run on a MacBookPro with M1 Max chip, current OS version is 14.4.1 (macOS Sonoma). The FasterSurfer version is the latest Stable branch https://github.com/Deep-MI/FastSurfer/tree/stable; the FreeSurfer Version is 7.4.1.

By checking the segmented images, I have noticed that six cortex regions are missing from the aparc.DKTatlas+aseg.deep.mgz, which can also be found in the sets.stats file

Screenshot 2024-04-25 at 11 23 52 AM

I also visually compared them using Freeview

Screenshot 2024-04-25 at 11 26 40 AM

So you can see that large 2028 ctx-rh_superiorfrontal region is mislabeled as the 2016 cox-rh-parapippocampal in aparc.DKTatlas+aseg.deep.mgz.

dkuegler commented 2 months ago

First off, thank you for the detailed report. We will try to recreate the issue you presented here.

m-reuter commented 2 months ago

Have you tried running FastSurfer (i) with run_fastsurfer --seg only and ii) the full pipeline ? I would not expect missing regions in either of those two options. I think that the problem comes from something you are doing differently from how FastSurfer does it.

gmeng92 commented 2 months ago

Have you tried running FastSurfer (i) with run_fastsurfer --seg only and ii) the full pipeline ? I would not expect missing regions in either of those two options. I think that the problem comes from something you are doing differently from how FastSurfer does it.

Yep. I tried the run_fastsurfer --seg only option but it keeps throwing out the syntax error.. But It should not make differences because this option basically runs the identically 3 steps as I mentioned. I also confirmed that I run with the same parameters as in run_fastsurfer.sh line 645-687

m-reuter commented 2 months ago

You are using the latest release version, right? And a native install or inside Docker? Our release (at least in docker) does not throw any syntax errors. That means that something is not setup correctly on your system and that may cause all kinds of problems.

gmeng92 commented 2 months ago

I used the latest release version and the native install. Besides that, I also tried to run the same python scripts in a GCP VM and the results are consistent with that on my MacPro. I will also re-setup and try the run_fastsurfer.sh script and get back to you once I figure it out.

gmeng92 commented 2 months ago

FYI. Thanks for your patience and response. I reset-up the native install and rerun the run_fastsurfer.sh by using

export PYTORCH_ENABLE_MPS_FALLBACK=1 input_dir=/Users/Projects/naip_data/Example_image output_dir=/Users/Projects/naip_data/Example_image/sh_run bash ./run_fastsurfer.sh --seg_only --t1 $input_dir/subject_140/example140_orig.mgz \ --sd $output_dir \ --sid subject_140 \ --seg_log $output_dir/subject_140/scripts/deep-seg.log \ --vox_size min

And the resulted aseg+DKT.stats indicates the same problem on the right hemisphere:

zainthecoder commented 2 months ago

I'm unable to reproduce the issue. Could you please create a Docker or Singularity container and try running the run_fastsurfer.sh script there?

In the meantime, could you share the deep-seg.log file here?

gmeng92 commented 2 months ago

deep-seg.log Here is the log file. I initially tried the Docker container but couldn't get it running there(see my previous post https://github.com/Deep-MI/FastSurfer/issues/379). That is why I used the native install and started from there. I will also keep debugging this issue. By the way, can you also share the log file as well? Appreciate for the response and help.

zainthecoder commented 1 month ago

Yes, Sure. Sharing my log file. deep-seg.log

m-reuter commented 4 weeks ago

@zainthecoder Note that this seems to be a native install on a modern Mac Book (with M chip). We need to try to replicate this on a similar Mac. Also note that #379 shows that docker does not work as expected on those Mac books (the reason is the ARM chips and our intel docker images will be very slow, as they also cannot use the GPU, but only CPU and need emulation). Which is why native install makes sense on the M-Chip Macs. It worked for me a while ago, and I could even make use of the GPU acceleration (segmentation in 2 minutes), but not sure about the more recent FastSurfer versions.

m-reuter commented 1 week ago

@gmeng92 I just merged #530 into the dev branch. This will make it easier to install and run FastSurfer on your M1 Mac. The only caveat is that the dev version of FastSurfer is not tested. But you could give it a try.

Also in order to replicate your results we would need your image and run it on a Mac and Linux. We do not expect regions to be missing and it indicates either a problem in the way you run it or a problem with the quality of the image (or simply out-of-distribution).

m-reuter commented 1 week ago

You could also try to run your setup on one of the tutorial images:

curl -k https://surfer.nmr.mgh.harvard.edu/pub/data/tutorial_data/buckner_data/tutorial_subjs/140/mri/orig.mgz -o "140_orig.mgz"

If it gives the same missing regions, it is your setup that has issues. If not, it is your image.