Missing error message (due to memory shortage)

[QunjunLIANG]

Question/Support Request

Hi FastSurfer experts,

I ran FastSurfer through Docker and attempted to obtain the cortical thickness in ROIs. However, the output directory of /subject/stats is empty, but the other outputs seem OK as what they are expected. I am not so familiar with FreeSurfer, so I want to ask if there is a way to get those stats report?

Environment

• FastSurfer Version: deemi/fastsurfer:gpu-v1.1.1
• OS: Ubuntu 20.04 LTS
• CPU: Intel Core i5-9400F CPU @2.90GHz
• GPU: Nvidia GTX 1660

Execution

docker run --gpus all -v /media/lqj/PUS4T/PickUpSam/NIFTI_BIDS/sub-01/anat:/data \ -v /media/lqj/PUS4T/PickUpSam/NIFTI_BIDS/sub-01/anat:/output \ -v /media/lqj/PUS4T/openNeuron_ds002748/derivatives:/fs_license \ --rm --user 1000 deepmi/fastsurfer:gpu-v1.1.1 \ --fs_license /fs_license/license.txt \ --t1 /data/sub-01_T1w.nii \ --sid sub-01 --sd /output \ --parallel --vol_segstats | tee /media/lqj/PUS4T/PickUpSam/NIFTI_BIDS/sub-01/anat/fs_log.txt

FastSurfer log - acquiring from the terminal

The running log was in attachment, please check.

FS_log.txt

[m-reuter]

Hi, you can try dropping the --vol_segstats flag (which is mainly used when running segmentation only, i.e. only the CNN, and you still want to get volume segmentation stats with the corpus callous added). Still that is probably not the problem here. Can you attach log file that is written under /stats/recon-surf.log ? thanks

[af-a]

Hi @QunjunLIANG.

Did you let FastSurfer run through to the end? Your log file indicates that it had not yet reached the anatomical statistics computation step. Once that is executed, you should find stats/{rh/lh}.aparc.mapped.stats (which is probably what you need).

Side note: Although you do not need --vol_segstats, it is strange that your stats directory is empty; in this case, it should contain aparc.DKTatlas+aseg.deep.volume.stats.

[m-reuter]

Maybe ran out of disk space?

[QunjunLIANG]

Thanks for your kind help.

I re-ran the FastSurfer docker this mooring with the recommended command (ignore --vol_segstats). But the output stats/ directory was still empty as that in yesterday.

The free space in the running disk is 1.6TB, and the total memory of the GPU is 6144MB. I uploaded all files in the script/ directory in attachment, hope these would be helpful to locate the problem.

Thank you for your time and support.

Attachment

1. The FastSurfer-running log obtained from the terminal. fs_log.txt

2. The document tree of the output directory, obtained with Linux 'tree' command. document_tree.txt

3. A zip contains all files in script/ directory. scripts.zip

[LeHenschel]

The log-file unfortunately does not contain an appropriate error message. Is this really the $SUBJECT/scripts/recon-surf.log file or the file you pipe the output to (with tee in the first command you provided)? The recon-surf.log should contain the error message.

Based on the file tree, it looks like the process is failing in the parallel hemi generation part, specifically when creating the aparc annotations (https://github.com/Deep-MI/FastSurfer/blob/v1.1.1/recon_surf/recon-surf.sh#L663, aparc.DKTatlas.mapped.prefix.annot is missing in label directory).

Without the error message, it is hard to know why it fails, so make sure you attached the recon-surf.log.

You can try to run the (potentially) offending command alone (if you have FreeSurfer installed and a local copy of the DKTatlaslookup.txt -> here I assume it is in FASTSURFER_HOME/recon_surf, https://github.com/Deep-MI/FastSurfer/blob/v1.1.1/recon_surf/rh.DKTatlaslookup.txt):

export SUBJECTS_DIR=/media/lqj/PUS4T/PickUpSam/NIFTI_BIDS/sub-01/anat
hemi=rh #or lh
subject=sub-01

mris_sample_parc -ct ${FREESURFER_HOME}/average/colortable_desikan_killiany.txt -file ${FASTSURFER_HOME}/recon_surf/$hemi.DKTatlaslookup.txt -projmm 0.6 -f 5  -surf white.preaparc $subject $hemi aparc.DKTatlas+aseg.orig.mgz aparc.DKTatlas.mapped.prefix.annot

[QunjunLIANG]

Thank you for your timely reply. I re-ran the docker without using the --parallel and the error report was finally out. The --parallel option may suppress the writing of the error on the log.

In the log, it wrote that "RMS increased, rejecting step" when entering MRISpositionSurface(). The recon-all log in subject/script/ was in attachment. Please let me know if there's any further information I can provide. recon-all.log

Thanks in advance.

[LeHenschel]

Ok, thanks. The recon-surf.log should contain an appropriate error message when sth fails in parallel mode. @AhmedFaisal95 @agirodi please look into this.

Do the segmentation (aseg.presurf.mgz), wm.mgz and original surface (lh.orig) look ok?

[QunjunLIANG]

Thank you for your timely reply @LeHenschel.

I checked the output images, including atlas segmentation and lh.orig with freeview, and they look fine in visual. Here's the screenshot of the freeview. Please check.

And I also uploaded the document tree in subject/ directory, hope this would help. document_tree.txt

All the best.

[LeHenschel]

Is it possible for you to share the image in question with us?

Based on the error, it looks like something is wrong with the surface generation. The segmentations look ok and the surface as well based on the screenshots. Is this the only subject that fails?

[agirodi]

From your provided logs it seems that the problem arises when the two sub-processes for each hemisphere gets interrupted/killed by something and the surface pipeline stops completely at that point (which is before the computation of the stats). That something, that sends a kill signal, is most probably your system, since our script doesn't seem to do it. It could be a lack of RAM or some limitations of resources by your system/administrator.

[QunjunLIANG]

Hi @LeHenschel and @agirodi ,

Thank you very much for your help. To fix the limitation of the RAM, I extended the SWAP from 2 GB to 8GB and reran FastSurfer with the same subject's data. As a result, the analysis finished without error reported in the recon-all.log. Please check down below: recon-all.log

However, there seems still some difference in the outputs, comparing with that generated by FreeSurfer. I attach the document tree from FastSurfer and FreeSurfer directory, please check down below:

document_tree_FastSurfer.txt

document_tree_FreeSurfer.txt

I called the FreeSurfer from the latest fMRIprep docker, version 7.2.0. Thus, I would like to ask if the gap between the outputs due to an potential error or the custom pipeline in the fMRIprep?

In addition, I upload the T1 image and its metadata (.json) on the Google Drive. Please download them with the following URL: https://drive.google.com/file/d/1ri5tDu3z9wAi5QO_kwFQLfXfkLXliWui/view?usp=sharing https://drive.google.com/file/d/1vhAQjAMd7sy9eIxoBlsZtctn9Z0A1YVg/view?usp=share_link

Thanks in advance.

Best, Qunjun

[LeHenschel]

I see. 2GB is too little to run FreeSurfer (https://surfer.nmr.mgh.harvard.edu/fswiki/SystemRequirements), so increasing it to 8 GB is a good call. Glad that it is working now.

With respect to the differences between FreeSurfer and FastSurfer directories: FastSurfer provides FreeSurfer like outputs. We only segment one atlas (DKTatlas) with the CNN. This is the reason for the differences you see in the directories. Are you interested in a specific file or atlas that is not present?

Also the output of the recon-surf.log is the important one. If that ran through without error, you can be sure that all files we provide are there.

[agirodi]

I run your image on our system and the output seems completely normal. Everything is there (I compared with the directory-tree of another image) and the segs/surfaces seem to be okay as well. Comparing your directory tree and ours did not raise any differences. tree_7.txt

[m-reuter]

https://github.com/Deep-MI/FastSurfer#system-requirements also describes our system requirements (in case someone else finds this thread). I think this issue is solved and can be closed, if not let us know.