search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
627 stars 290 forks source link

recon-all randomly fails for the exact same data (fmriprep-docker) #3220

Closed isacdaavid closed 7 months ago

isacdaavid commented 7 months ago

What happened?

We are running fmriprep within docker as non-root users for a pretty standard subject database (rs-fMRI, T1, T2). It's working great except for the fact that each user takes 2 or 3 attempts of the same command and cleaning-up of the failed derivatives folder before success.

We think we have ruled out concurrency issues. Even if run serially, freesurfer sometimes fails and sometimes doesn't for exactly the same data and invocation (and it would be great to be able to process many users in parallel anyway, but first we want to figure out how to increase our success rate).

Are we doing something wrong? The logs are a bit arcane to us.

What command did you use?

fmriprep-docker /Volumes/NMorphCHfMRIprep/ /Volumes/NMorphCHderivatives --participant-label sub-CH8461a --fs-license-file license.txt --cifti-output

RUNNING: docker run --rm -e DOCKER_VERSION_8395080871=24.0.7 -it -v /home/isaac/license.txt:/opt/freesurfer/license.txt:ro -v /Volumes/NMorphCHfMRIprep:/data:ro -v /Volumes/NMorphCHderivatives:/out nipreps/fmriprep:23.2.0 /data /out participant --participant-label sub-CH8461a --cifti-output

What version of fMRIPrep are you running?

fMRIPrep v23.2.0

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Node Name: _autorecon_surfs1

File: /out/sub-CH8461a/log/20240205-165545_1f777ac0-bdfd-41aa-96c9-423701a827c6/crash-20240205-171357-root-_autorecon_surfs1-7a3740e3-9729-4aad-8d00-4721c5f5a0dd.txt
Working Directory: /tmp/work/fmriprep_23_2_wf/sub_CH8461a_wf/anat_fit_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs1
Inputs:

    FLAIR_file:
    T1_files:
    T2_file:
    args:
    big_ventricles:
    brainstem:
    directive: autorecon-hemi
    environ: {}
    expert:
    flags: ['-noparcstats', '-noparcstats2', '-noparcstats3', '-nohyporelabel', '-nobalabels']
    hemi: rh
    hippocampal_subfields_T1:
    hippocampal_subfields_T2:
    hires:
    mprage:
    mri_aparc2aseg:
    mri_ca_label:
    mri_ca_normalize:
    mri_ca_register:
    mri_edit_wm_with_aseg:
    mri_em_register:
    mri_fill:
    mri_mask:
    mri_normalize:
    mri_pretess:
    mri_remove_neck:
    mri_segment:
    mri_segstats:
    mri_tessellate:
    mri_watershed:
    mris_anatomical_stats:
    mris_ca_label:
    mris_fix_topology:
    mris_inflate:
    mris_make_surfaces:
    mris_register:
    mris_smooth:
    mris_sphere:
    mris_surf2vol:
    mrisp_paint:
    openmp: 8
    parallel:
    steps:
    subject_id: sub-CH8461a
    subjects_dir: /out/sourcedata/freesurfer
    talairach:
    use_FLAIR:
    use_T2:
    xopts:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _autorecon_surfs1.

Cmdline:
    recon-all -autorecon-hemi rh -noparcstats -noparcstats2 -noparcstats3 -nohyporelabel -nobalabels -rh-only -openmp 8 -subjid sub-CH8461a -sd /out/sourcedata/freesurfer 
Stdout:
    fs-check-version --s sub-CH8461a --o /tmp/tmp.P514n0
    Mon Feb  5 17:11:37 UTC 2024

    setenv SUBJECTS_DIR /out/sourcedata/freesurfer
    cd /tmp/work/fmriprep_23_2_wf/sub_CH8461a_wf/anat_fit_wf/surface_recon_wf/autorecon_resume_wf/autorecon_surfs/mapflow/_autorecon_surfs1
    /opt/freesurfer/bin/fs-check-version --s sub-CH8461a --o /tmp/tmp.P514n0
    -rwxrwxr-x 1 root root 18565 Aug  4  2022 /opt/freesurfer/bin/fs-check-version

    freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275
    $Id$
    Linux 479c361707c2 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
    pid 872
    Current FS Version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275
    bstampfile exists /out/sourcedata/freesurfer/sub-CH8461a/scripts/build-stamp.txt
    Subject FS Version: freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275
    No constraints on version because REQ=UnSet and FsVerFile=NotThere
    #@#% fs-check-version match = 1
    fs-check-version Done
    INFO: SUBJECTS_DIR is /out/sourcedata/freesurfer
    Actual FREESURFER_HOME /opt/freesurfer
    Linux 479c361707c2 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
    /out/sourcedata/freesurfer/sub-CH8461a/mri/transforms /out/sourcedata/freesurfer/sub-CH8461a 
    /out/sourcedata/freesurfer/sub-CH8461a 
    #--------------------------------------------
    #@# Tessellate rh Mon Feb  5 17:11:40 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8461a/scripts

     mri_pretess ../mri/filled.mgz 127 ../mri/norm.mgz ../mri/filled-pretess127.mgz 

    Iteration Number : 1
    pass   1 (xy+):   1 found -   1 modified     |    TOTAL:   1
    pass   2 (xy+):   0 found -   1 modified     |    TOTAL:   1
    pass   1 (xy-):   1 found -   1 modified     |    TOTAL:   2
    pass   2 (xy-):   0 found -   1 modified     |    TOTAL:   2
    pass   1 (yz+):   5 found -   5 modified     |    TOTAL:   7
    pass   2 (yz+):   0 found -   5 modified     |    TOTAL:   7
    pass   1 (yz-):   8 found -   8 modified     |    TOTAL:  15
    pass   2 (yz-):   0 found -   8 modified     |    TOTAL:  15
    pass   1 (xz+):   0 found -   0 modified     |    TOTAL:  15
    pass   1 (xz-):   0 found -   0 modified     |    TOTAL:  15
    Iteration Number : 1
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 1
    pass   1 (++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (--):   2 found -   2 modified     |    TOTAL:   2
    pass   2 (--):   0 found -   2 modified     |    TOTAL:   2
    pass   1 (-+):   0 found -   0 modified     |    TOTAL:   2
    Iteration Number : 2
    pass   1 (xy+):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (xy-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (yz+):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (yz-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (xz+):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (xz-):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 2
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 2
    pass   1 (++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (--):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (-+):   0 found -   0 modified     |    TOTAL:   0

    Total Number of Modified Voxels = 17 (out of 265931: 0.006393)
    Ambiguous edge configurations... 

    mri_pretess done

     mri_tessellate ../mri/filled-pretess127.mgz 127 ../surf/rh.orig.nofix 

    7.3.2
      7.3.2
    slice 40: 547 vertices, 608 faces
    slice 50: 4608 vertices, 4820 faces
    slice 60: 12630 vertices, 12955 faces
    slice 70: 22689 vertices, 23039 faces
    slice 80: 32873 vertices, 33197 faces
    slice 90: 42737 vertices, 43030 faces
    slice 100: 53340 vertices, 53697 faces
    slice 110: 63561 vertices, 63910 faces
    slice 120: 73411 vertices, 73747 faces
    slice 130: 82737 vertices, 83101 faces
    slice 140: 91815 vertices, 92110 faces
    slice 150: 99989 vertices, 100261 faces
    slice 160: 107195 vertices, 107444 faces
    slice 170: 113118 vertices, 113290 faces
    slice 180: 117787 vertices, 117954 faces
    slice 190: 121449 vertices, 121560 faces
    slice 200: 122360 vertices, 122362 faces
    slice 210: 122360 vertices, 122362 faces
    slice 220: 122360 vertices, 122362 faces
    slice 230: 122360 vertices, 122362 faces
    slice 240: 122360 vertices, 122362 faces
    slice 250: 122360 vertices, 122362 faces
    using the conformed surface RAS to save vertex points...
    writing ../surf/rh.orig.nofix
    using vox2ras matrix:
    -1.00000   0.00000   0.00000   128.00000;
     0.00000   0.00000   1.00000  -128.00000;
     0.00000  -1.00000   0.00000   128.00000;
     0.00000   0.00000   0.00000   1.00000;

     rm -f ../mri/filled-pretess127.mgz 

     mris_extract_main_component ../surf/rh.orig.nofix ../surf/rh.orig.nofix 

    counting number of connected components...
       122360 voxel in cpt #1: X=-2 [v=122360,e=367086,f=244724] located at (26.243225, -19.038059, 34.952198)
    For the whole surface: X=-2 [v=122360,e=367086,f=244724]
    One single component has been found
    nothing to do
    done

    #--------------------------------------------
    #@# Smooth1 rh Mon Feb  5 17:11:44 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8461a/scripts

     mris_smooth -nw -seed 1234 ../surf/rh.orig.nofix ../surf/rh.smoothwm.nofix 

    setting seed for random number generator to 1234
    smoothing surface tessellation for 10 iterations...
    smoothing complete - recomputing first and second fundamental forms...
    #--------------------------------------------
    #@# Inflation1 rh Mon Feb  5 17:11:47 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8461a/scripts

     mris_inflate -no-save-sulc ../surf/rh.smoothwm.nofix ../surf/rh.inflated.nofix 

    Not saving sulc
    Reading ../surf/rh.smoothwm.nofix
    avg radius = 45.6 mm, total surface area = 66569 mm^2

    step 000: RMS=0.144 (target=0.015)   
    step 005: RMS=0.107 (target=0.015)   
    step 010: RMS=0.078 (target=0.015)   
    step 015: RMS=0.064 (target=0.015)   
    step 020: RMS=0.054 (target=0.015)   
    step 025: RMS=0.046 (target=0.015)   
    step 030: RMS=0.041 (target=0.015)   
    step 035: RMS=0.037 (target=0.015)   
    step 040: RMS=0.034 (target=0.015)   
    step 045: RMS=0.031 (target=0.015)   
    step 050: RMS=0.029 (target=0.015)   
    step 055: RMS=0.029 (target=0.015)   
    step 060: RMS=0.029 (target=0.015)   writing inflated surface to ../surf/rh.inflated.nofix
    inflation took 0.2 minutes

    inflation complete.
    Not saving sulc
    mris_inflate utimesec    104.086666
    mris_inflate stimesec    5.083934
    mris_inflate ru_maxrss   195056
    mris_inflate ru_ixrss    0
    mris_inflate ru_idrss    0
    mris_inflate ru_isrss    0
    mris_inflate ru_minflt   666856
    mris_inflate ru_majflt   0
    mris_inflate ru_nswap    0
    mris_inflate ru_inblock  0
    mris_inflate ru_oublock  8608
    mris_inflate ru_msgsnd   0
    mris_inflate ru_msgrcv   0
    mris_inflate ru_nsignals 0
    mris_inflate ru_nvcsw    3045
    mris_inflate ru_nivcsw   384
    #--------------------------------------------
    #@# QSphere rh Mon Feb  5 17:12:01 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8461a/scripts

     mris_sphere -q -p 6 -a 128 -seed 1234 ../surf/rh.inflated.nofix ../surf/rh.qsphere.nofix 

    doing quick spherical unfolding.
    limitting unfolding to 6 passes
    using n_averages = 128
    setting seed for random number genererator to 1234
    version: 7.3.2
    available threads: 8
    scaling brain by 0.324...
    inflating...
    projecting onto sphere...
    surface projected - minimizing metric distortion...
    vertex spacing 1.06 +- 0.57 (0.00-->7.03) (max @ vno 93330 --> 93359)
    face area 0.03 +- 0.03 (-0.06-->0.68)
    Entering MRISinflateToSphere()
    inflating to sphere (rms error < 2.00)
    000: dt: 0.0000, rms radial error=176.717, avgs=0
    005/300: dt: 0.9000, rms radial error=176.457, avgs=0
    010/300: dt: 0.9000, rms radial error=175.899, avgs=0
    015/300: dt: 0.9000, rms radial error=175.165, avgs=0
    020/300: dt: 0.9000, rms radial error=174.329, avgs=0
    025/300: dt: 0.9000, rms radial error=173.436, avgs=0
    030/300: dt: 0.9000, rms radial error=172.514, avgs=0
    035/300: dt: 0.9000, rms radial error=171.575, avgs=0
    040/300: dt: 0.9000, rms radial error=170.628, avgs=0
    045/300: dt: 0.9000, rms radial error=169.677, avgs=0
    050/300: dt: 0.9000, rms radial error=168.728, avgs=0
    055/300: dt: 0.9000, rms radial error=167.783, avgs=0
    060/300: dt: 0.9000, rms radial error=166.842, avgs=0
    065/300: dt: 0.9000, rms radial error=165.906, avgs=0
    070/300: dt: 0.9000, rms radial error=164.975, avgs=0
    075/300: dt: 0.9000, rms radial error=164.049, avgs=0
    080/300: dt: 0.9000, rms radial error=163.128, avgs=0
    085/300: dt: 0.9000, rms radial error=162.211, avgs=0
    090/300: dt: 0.9000, rms radial error=161.300, avgs=0
    095/300: dt: 0.9000, rms radial error=160.394, avgs=0
    100/300: dt: 0.9000, rms radial error=159.493, avgs=0
    105/300: dt: 0.9000, rms radial error=158.597, avgs=0
    110/300: dt: 0.9000, rms radial error=157.706, avgs=0
    115/300: dt: 0.9000, rms radial error=156.820, avgs=0
    120/300: dt: 0.9000, rms radial error=155.939, avgs=0
    125/300: dt: 0.9000, rms radial error=155.063, avgs=0
    130/300: dt: 0.9000, rms radial error=154.192, avgs=0
    135/300: dt: 0.9000, rms radial error=153.325, avgs=0
    140/300: dt: 0.9000, rms radial error=152.463, avgs=0
    145/300: dt: 0.9000, rms radial error=151.606, avgs=0
    150/300: dt: 0.9000, rms radial error=150.754, avgs=0
    155/300: dt: 0.9000, rms radial error=149.906, avgs=0
    160/300: dt: 0.9000, rms radial error=149.063, avgs=0
    165/300: dt: 0.9000, rms radial error=148.225, avgs=0
    170/300: dt: 0.9000, rms radial error=147.391, avgs=0
    175/300: dt: 0.9000, rms radial error=146.562, avgs=0
    180/300: dt: 0.9000, rms radial error=145.738, avgs=0
    185/300: dt: 0.9000, rms radial error=144.918, avgs=0
    190/300: dt: 0.9000, rms radial error=144.103, avgs=0
    195/300: dt: 0.9000, rms radial error=143.292, avgs=0
    200/300: dt: 0.9000, rms radial error=142.486, avgs=0
    205/300: dt: 0.9000, rms radial error=141.684, avgs=0
    210/300: dt: 0.9000, rms radial error=140.887, avgs=0
    215/300: dt: 0.9000, rms radial error=140.094, avgs=0
    220/300: dt: 0.9000, rms radial error=139.305, avgs=0
    225/300: dt: 0.9000, rms radial error=138.521, avgs=0
    230/300: dt: 0.9000, rms radial error=137.741, avgs=0
    235/300: dt: 0.9000, rms radial error=136.965, avgs=0
    240/300: dt: 0.9000, rms radial error=136.194, avgs=0
    245/300: dt: 0.9000, rms radial error=135.427, avgs=0
    250/300: dt: 0.9000, rms radial error=134.664, avgs=0
    255/300: dt: 0.9000, rms radial error=133.905, avgs=0
    260/300: dt: 0.9000, rms radial error=133.151, avgs=0
    265/300: dt: 0.9000, rms radial error=132.401, avgs=0
    270/300: dt: 0.9000, rms radial error=131.655, avgs=0
    275/300: dt: 0.9000, rms radial error=130.913, avgs=0
    280/300: dt: 0.9000, rms radial error=130.175, avgs=0
    285/300: dt: 0.9000, rms radial error=129.441, avgs=0
    290/300: dt: 0.9000, rms radial error=128.712, avgs=0
    295/300: dt: 0.9000, rms radial error=127.986, avgs=0
    300/300: dt: 0.9000, rms radial error=127.265, avgs=0

    spherical inflation complete.
    epoch 1 (K=10.0), pass 1, starting sse = 13986.56
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    pass 1 complete, delta sse/iter = 0.00/13 = 0.00029
    epoch 2 (K=40.0), pass 1, starting sse = 2159.38
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    pass 1 complete, delta sse/iter = 0.01/13 = 0.00053
    epoch 3 (K=160.0), pass 1, starting sse = 173.83
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    pass 1 complete, delta sse/iter = 0.18/18 = 0.01016
    epoch 4 (K=640.0), pass 1, starting sse = 4.82
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    taking momentum steps...
    pass 1 complete, delta sse/iter = 0.30/36 = 0.00825
    final distance error %100000.00
    writing spherical brain to ../surf/rh.qsphere.nofix
    spherical transformation took 0.0223 hours
    FSRUNTIME@ mris_sphere  0.0223 hours 8 threads
    #VMPC# mris_sphere VmPeak  948024
    mris_sphere done
    #@# Fix Topology rh Mon Feb  5 17:13:22 UTC 2024

     mris_fix_topology -mgz -sphere qsphere.nofix -inflated inflated.nofix -orig orig.nofix -out orig.premesh -ga -seed 1234 sub-CH8461a rh 

    reading spherical homeomorphism from 'qsphere.nofix'
    reading inflated coordinates from 'inflated.nofix'
    reading original coordinates from 'orig.nofix'
    using genetic algorithm with optimized parameters
    setting seed for random number genererator to 1234

    *************************************************************
    Topology Correction Parameters
    retessellation mode:           genetic search
    number of patches/generation : 10
    number of generations :        10
    surface mri loglikelihood coefficient :         1.0
    volume mri loglikelihood coefficient :          10.0
    normal dot loglikelihood coefficient :          1.0
    quadratic curvature loglikelihood coefficient : 1.0
    volume resolution :                             2
    eliminate vertices during search :              1
    initial patch selection :                       1
    select all defect vertices :                    0
    ordering dependant retessellation:              0
    use precomputed edge table :                    0
    smooth retessellated patch :                    2
    match retessellated patch :                     1
    verbose mode :                                  0

    *************************************************************
    INFO: assuming .mgz format
    writing corrected surface to 'orig.premesh'
    7.3.2
      7.3.2
    before topology correction, eno=-2 (nv=122360, nf=244724, ne=367086, g=2)
    using quasi-homeomorphic spherical map to tessellate cortical surface...

    Correction of the Topology
    Finding true center and radius of Spherical Surface...done
    Surface centered at (0,0,0) with radius 100.0 in 8 iterations
    marking ambiguous vertices...
    1153 ambiguous faces found in tessellation
    segmenting defects...
    12 defects found, arbitrating ambiguous regions...
    analyzing neighboring defects...
    12 defects to be corrected 
    0 vertices coincident
    reading input surface /out/sourcedata/freesurfer/sub-CH8461a/surf/rh.qsphere.nofix...
    reading brain volume from brain...
    reading wm segmentation from wm...
    Reading original properties of orig.nofix
    Reading vertex positions of inflated.nofix
    Computing Initial Surface Statistics
          -face       loglikelihood: -9.3519  (-4.6760)
          -vertex     loglikelihood: -6.0938  (-3.0469)
          -normal dot loglikelihood: -3.5283  (-3.5283)
          -quad curv  loglikelihood: -5.8521  (-2.9260)
          Total Loglikelihood : -24.8261
    CORRECTING DEFECT 0 (vertices=10, convex hull=28, v0=30655)
    After retessellation of defect 0 (v0=30655), euler #=-9 (121685,364874,243180) : difference with theory (-9) = 0 
    CORRECTING DEFECT 1 (vertices=11, convex hull=32, v0=43391)
    After retessellation of defect 1 (v0=43391), euler #=-8 (121688,364893,243197) : difference with theory (-8) = 0 
    CORRECTING DEFECT 2 (vertices=30, convex hull=48, v0=49662)
    After retessellation of defect 2 (v0=49662), euler #=-7 (121701,364950,243242) : difference with theory (-7) = 0 
    CORRECTING DEFECT 3 (vertices=5, convex hull=14, v0=54906)
    After retessellation of defect 3 (v0=54906), euler #=-6 (121702,364956,243248) : difference with theory (-6) = 0 
    CORRECTING DEFECT 4 (vertices=56, convex hull=28, v0=61538)
    After retessellation of defect 4 (v0=61538), euler #=-5 (121706,364977,243266) : difference with theory (-5) = 0 
    CORRECTING DEFECT 5 (vertices=7, convex hull=16, v0=62358)
    After retessellation of defect 5 (v0=62358), euler #=-4 (121707,364982,243271) : difference with theory (-4) = 0 
    CORRECTING DEFECT 6 (vertices=22, convex hull=19, v0=74401)
    After retessellation of defect 6 (v0=74401), euler #=-3 (121710,364999,243286) : difference with theory (-3) = 0 
    CORRECTING DEFECT 7 (vertices=49, convex hull=43, v0=80725)
    After retessellation of defect 7 (v0=80725), euler #=-2 (121713,365024,243309) : difference with theory (-2) = 0 
    CORRECTING DEFECT 8 (vertices=25, convex hull=61, v0=85530)
    After retessellation of defect 8 (v0=85530), euler #=-1 (121727,365089,243361) : difference with theory (-1) = 0 
    CORRECTING DEFECT 9 (vertices=6, convex hull=25, v0=86493)
    After retessellation of defect 9 (v0=86493), euler #=0 (121728,365098,243370) : difference with theory (0) = 0 
    CORRECTING DEFECT 10 (vertices=391, convex hull=134, v0=93482)
    After retessellation of defect 10 (v0=93482), euler #=1 (121788,365342,243555) : difference with theory (1) = 0 
    CORRECTING DEFECT 11 (vertices=64, convex hull=44, v0=106448)
    After retessellation of defect 11 (v0=106448), euler #=2 (121799,365391,243594) : difference with theory (2) = 0 
    computing original vertex metric properties...
    storing new metric properties...
    computing tessellation statistics...
    vertex spacing 0.89 +- 0.20 (0.06-->3.68) (max @ vno 50757 --> 51852)
    face area -nan +- -nan (1000.00-->-1.00)
    performing soap bubble on retessellated vertices for 0 iterations...
    vertex spacing 0.89 +- 0.20 (0.06-->3.68) (max @ vno 50757 --> 51852)
    face area -nan +- -nan (1000.00-->-1.00)
    tessellation finished, orienting corrected surface...
    44 mutations (40.4%), 65 crossovers (59.6%), 7 vertices were eliminated
    building final representation...
    561 vertices and 0 faces have been removed from triangulation
    after topology correction, eno=2 (nv=121799, nf=243594, ne=365391, g=0)
    writing corrected surface to /out/sourcedata/freesurfer/sub-CH8461a/surf/rh.orig.premesh...

    0.000 % of the vertices (0 vertices) exhibit an orientation change
    removing intersecting faces
    000: 33 intersecting
    001: 3 intersecting
    terminating search with 0 intersecting
    topology fixing took 0.5 minutes
    FSRUNTIME@ mris_fix_topology rh  0.0090 hours 8 threads
    #VMPC# mris_fix_topology VmPeak  1512300

     mris_euler_number ../surf/rh.orig.premesh 

    euler # = v-e+f = 2g-2: 121799 - 365391 + 243594 = 2 --> 0 holes
          F =2V-4:          243594 = 243598-4 (0)
          2E=3F:            730782 = 730782 (0)

    total defect index = 0
    /out/sourcedata/freesurfer/sub-CH8461a/scripts/defect2seg.log: No such file or directory.
    Linux 479c361707c2 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

    recon-all -s sub-CH8461a exited with ERRORS at Mon Feb  5 17:13:55 UTC 2024

    For more details, see the log file /out/sourcedata/freesurfer/sub-CH8461a/scripts/recon-all-rh.log
    To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Stderr:
    cat: standard output: Stale file handle
    cat: standard output: Stale file handle
Traceback:
    RuntimeError: subprocess exited with code 1.

Additional information / screenshots

sourcedata/freesurfer/sub-CH8461a/scripts/recon-all-rh.log seems to contain the same data as the previous log.

isacdaavid commented 7 months ago

Sometimes we get a different error. It seems like cp fails copying a file (filled.mgz), also during the recon-all phase of freesurfer:

File: /out/sub-CH8698b/log/20240205-022330_fd0d60cc-3c45-4f6f-9346-9bb73e5d702d/crash-20240205-032115-root-autorecon2_vol-0482c74c-54d6-4802-a38b-5041180a18fe.txt
Working Directory: /tmp/work/fmriprep_23_2_wf/sub_CH8698b_wf/anat_fit_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol
Inputs:

    FLAIR_file:
    T1_files:
    T2_file:
    args:
    big_ventricles:
    brainstem:
    directive: autorecon2-volonly
    environ: {}
    expert:
    flags:
    hemi:
    hippocampal_subfields_T1:
    hippocampal_subfields_T2:
    hires:
    mprage:
    mri_aparc2aseg:
    mri_ca_label:
    mri_ca_normalize:
    mri_ca_register:
    mri_edit_wm_with_aseg:
    mri_em_register:
    mri_fill:
    mri_mask:
    mri_normalize:
    mri_pretess:
    mri_remove_neck:
    mri_segment:
    mri_segstats:
    mri_tessellate:
    mri_watershed:
    mris_anatomical_stats:
    mris_ca_label:
    mris_fix_topology:
    mris_inflate:
    mris_make_surfaces:
    mris_register:
    mris_smooth:
    mris_sphere:
    mris_surf2vol:
    mrisp_paint:
    openmp: 8
    parallel:
    steps:
    subject_id: recon_all
    subjects_dir:
    talairach:
    use_FLAIR:
    use_T2:
    xopts:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon2_vol.

Cmdline:
    recon-all -autorecon2-volonly -openmp 8 -subjid sub-CH8698b -sd /out/sourcedata/freesurfer -nogcareg -nocanorm -nocareg
Stdout:
    fs-check-version --s sub-CH8698b --o /tmp/tmp.zS155I
    Mon Feb  5 02:39:30 UTC 2024

    setenv SUBJECTS_DIR /out/sourcedata/freesurfer
    cd /tmp/work/fmriprep_23_2_wf/sub_CH8698b_wf/anat_fit_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol
    /opt/freesurfer/bin/fs-check-version --s sub-CH8698b --o /tmp/tmp.zS155I
    -rwxrwxr-x 1 root root 18565 Aug  4  2022 /opt/freesurfer/bin/fs-check-version

    freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275
    $Id$
    Linux a6080e14df56 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
    pid 698
    Current FS Version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275
    bstampfile exists /out/sourcedata/freesurfer/sub-CH8698b/scripts/build-stamp.txt
    Subject FS Version: freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275
    No constraints on version because REQ=UnSet and FsVerFile=NotThere
    #@#% fs-check-version match = 1
    fs-check-version Done
    INFO: SUBJECTS_DIR is /out/sourcedata/freesurfer
    Actual FREESURFER_HOME /opt/freesurfer
    -rw-rw-r-- 1 root root 246032 Jan 29 11:51 /out/sourcedata/freesurfer/sub-CH8698b/scripts/recon-all.log
    Linux a6080e14df56 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
    /out/sourcedata/freesurfer/sub-CH8698b/mri/transforms /out/sourcedata/freesurfer/sub-CH8698b 
    /out/sourcedata/freesurfer/sub-CH8698b 
    #--------------------------------------
    #@# SubCort Seg Mon Feb  5 02:39:32 UTC 2024

     mri_ca_label -relabel_unlikely 9 .3 -prior 0.5 -align norm.mgz transforms/talairach.m3z /opt/freesurfer/average/RB_all_2020-01-02.gca aseg.auto_noCCseg.mgz 

    sysname  Linux
    hostname a6080e14df56
    machine  x86_64

    setenv SUBJECTS_DIR /out/sourcedata/freesurfer
    cd /out/sourcedata/freesurfer/sub-CH8698b/mri
    mri_ca_label -relabel_unlikely 9 .3 -prior 0.5 -align norm.mgz transforms/talairach.m3z /opt/freesurfer/average/RB_all_2020-01-02.gca aseg.auto_noCCseg.mgz 

    relabeling unlikely voxels with window_size = 9 and prior threshold 0.30
    using Gibbs prior factor = 0.500
    renormalizing sequences with structure alignment, equivalent to:
        -renormalize
        -renormalize_mean 0.500
        -regularize 0.500

    == Number of threads available to for OpenMP = 8 == 
    reading 1 input volumes
    reading classifier array from /opt/freesurfer/average/RB_all_2020-01-02.gca
    reading input volume from norm.mgz
    average std[0] = 7.2
    reading transform from transforms/talairach.m3z
    setting orig areas to linear transform determinant scaled 5.33
    Atlas used for the 3D morph was /opt/freesurfer/average/RB_all_2020-01-02.gca
    average std = 7.2   using min determinant for regularization = 5.2
    0 singular and 0 ill-conditioned covariance matrices regularized
    labeling volume...
    renormalizing by structure alignment....
    renormalizing input #0
    gca peak = 0.15521 (20)
    mri peak = 0.10044 (12)
    Left_Lateral_Ventricle (4): linear fit = 0.70 x + 0.0 (916 voxels, overlap=0.646)
    Left_Lateral_Ventricle (4): linear fit = 0.70 x + 0.0 (916 voxels, peak = 14), gca=14.1
    gca peak = 0.20380 (13)
    mri peak = 0.14649 (18)
    Right_Lateral_Ventricle (43): linear fit = 1.20 x + 0.0 (1514 voxels, overlap=0.743)
    Right_Lateral_Ventricle (43): linear fit = 1.20 x + 0.0 (1514 voxels, peak = 16), gca=15.5
    gca peak = 0.26283 (96)
    mri peak = 0.21672 (98)
    Right_Pallidum (52): linear fit = 1.02 x + 0.0 (646 voxels, overlap=0.932)
    Right_Pallidum (52): linear fit = 1.02 x + 0.0 (646 voxels, peak = 98), gca=98.4
    gca peak = 0.15814 (97)
    mri peak = 0.23288 (97)
    Left_Pallidum (13): linear fit = 1.00 x + 0.0 (657 voxels, overlap=0.947)
    Left_Pallidum (13): linear fit = 1.00 x + 0.0 (657 voxels, peak = 97), gca=96.5
    gca peak = 0.27624 (56)
    mri peak = 0.21287 (67)
    Right_Hippocampus (53): linear fit = 1.21 x + 0.0 (823 voxels, overlap=0.013)
    Right_Hippocampus (53): linear fit = 1.21 x + 0.0 (823 voxels, peak = 67), gca=67.5
    gca peak = 0.28723 (59)
    mri peak = 0.13483 (65)
    Left_Hippocampus (17): linear fit = 1.12 x + 0.0 (751 voxels, overlap=0.136)
    Left_Hippocampus (17): linear fit = 1.12 x + 0.0 (751 voxels, peak = 66), gca=65.8
    gca peak = 0.07623 (103)
    mri peak = 0.26841 (106)
    Right_Cerebral_White_Matter (41): linear fit = 1.01 x + 0.0 (32829 voxels, overlap=0.457)
    Right_Cerebral_White_Matter (41): linear fit = 1.01 x + 0.0 (32829 voxels, peak = 105), gca=104.5
    gca peak = 0.07837 (105)
    mri peak = 0.21465 (105)
    Left_Cerebral_White_Matter (2): linear fit = 1.00 x + 0.0 (35531 voxels, overlap=0.551)
    Left_Cerebral_White_Matter (2): linear fit = 1.00 x + 0.0 (35531 voxels, peak = 104), gca=104.5
    gca peak = 0.10165 (58)
    mri peak = 0.04782 (66)
    Left_Cerebral_Cortex (3): linear fit = 1.12 x + 0.0 (22966 voxels, overlap=0.645)
    Left_Cerebral_Cortex (3): linear fit = 1.12 x + 0.0 (22966 voxels, peak = 65), gca=64.7
    gca peak = 0.11113 (58)
    mri peak = 0.05009 (63)
    Right_Cerebral_Cortex (42): linear fit = 1.10 x + 0.0 (20600 voxels, overlap=0.669)
    Right_Cerebral_Cortex (42): linear fit = 1.10 x + 0.0 (20600 voxels, peak = 64), gca=63.5
    gca peak = 0.27796 (67)
    mri peak = 0.13764 (78)
    Right_Caudate (50): linear fit = 1.12 x + 0.0 (819 voxels, overlap=0.056)
    Right_Caudate (50): linear fit = 1.12 x + 0.0 (819 voxels, peak = 75), gca=75.4
    gca peak = 0.14473 (69)
    mri peak = 0.16890 (77)
    Left_Caudate (11): linear fit = 1.00 x + 0.0 (821 voxels, overlap=0.674)
    Left_Caudate (11): linear fit = 1.00 x + 0.0 (821 voxels, peak = 69), gca=69.0
    gca peak = 0.14301 (56)
    mri peak = 0.07530 (66)
    Left_Cerebellum_Cortex (8): linear fit = 1.15 x + 0.0 (22259 voxels, overlap=0.246)
    Left_Cerebellum_Cortex (8): linear fit = 1.15 x + 0.0 (22259 voxels, peak = 65), gca=64.7
    gca peak = 0.14610 (55)
    mri peak = 0.07220 (68)
    Right_Cerebellum_Cortex (47): linear fit = 1.21 x + 0.0 (23168 voxels, overlap=0.164)
    Right_Cerebellum_Cortex (47): linear fit = 1.21 x + 0.0 (23168 voxels, peak = 66), gca=66.3
    gca peak = 0.16309 (85)
    mri peak = 0.25949 (91)
    Left_Cerebellum_White_Matter (7): linear fit = 1.08 x + 0.0 (5523 voxels, overlap=0.018)
    Left_Cerebellum_White_Matter (7): linear fit = 1.08 x + 0.0 (5523 voxels, peak = 91), gca=91.4
    gca peak = 0.15172 (84)
    mri peak = 0.17737 (90)
    Right_Cerebellum_White_Matter (46): linear fit = 1.09 x + 0.0 (5334 voxels, overlap=0.068)
    Right_Cerebellum_White_Matter (46): linear fit = 1.09 x + 0.0 (5334 voxels, peak = 91), gca=91.1
    gca peak = 0.30461 (58)
    mri peak = 0.16252 (72)
    Left_Amygdala (18): linear fit = 1.24 x + 0.0 (378 voxels, overlap=0.036)
    Left_Amygdala (18): linear fit = 1.24 x + 0.0 (378 voxels, peak = 72), gca=71.6
    gca peak = 0.32293 (57)
    mri peak = 0.14222 (70)
    Right_Amygdala (54): linear fit = 1.23 x + 0.0 (450 voxels, overlap=0.040)
    Right_Amygdala (54): linear fit = 1.23 x + 0.0 (450 voxels, peak = 70), gca=69.8
    gca peak = 0.11083 (90)
    mri peak = 0.08966 (92)
    Left_Thalamus (10): linear fit = 1.03 x + 0.0 (4239 voxels, overlap=0.841)
    Left_Thalamus (10): linear fit = 1.03 x + 0.0 (4239 voxels, peak = 93), gca=93.1
    gca peak = 0.11393 (83)
    mri peak = 0.09267 (94)
    Right_Thalamus (49): linear fit = 1.11 x + 0.0 (3616 voxels, overlap=0.609)
    Right_Thalamus (49): linear fit = 1.11 x + 0.0 (3616 voxels, peak = 92), gca=91.7
    gca peak = 0.08575 (81)
    mri peak = 0.13502 (83)
    Left_Putamen (12): linear fit = 1.03 x + 0.0 (1573 voxels, overlap=0.577)
    Left_Putamen (12): linear fit = 1.03 x + 0.0 (1573 voxels, peak = 84), gca=83.8
    gca peak = 0.08618 (78)
    mri peak = 0.11726 (83)
    Right_Putamen (51): linear fit = 1.07 x + 0.0 (1825 voxels, overlap=0.761)
    Right_Putamen (51): linear fit = 1.07 x + 0.0 (1825 voxels, peak = 83), gca=83.1
    gca peak = 0.08005 (78)
    mri peak = 0.10605 (90)
    Brain_Stem (16): linear fit = 1.13 x + 0.0 (10876 voxels, overlap=0.366)
    Brain_Stem (16): linear fit = 1.13 x + 0.0 (10876 voxels, peak = 89), gca=88.5
    gca peak = 0.12854 (88)
    mri peak = 0.13885 (100)
    Right_VentralDC (60): linear fit = 1.11 x + 0.0 (1312 voxels, overlap=0.018)
    Right_VentralDC (60): linear fit = 1.11 x + 0.0 (1312 voxels, peak = 97), gca=97.2
    gca peak = 0.15703 (87)
    mri peak = 0.16600 (100)
    Left_VentralDC (28): linear fit = 1.11 x + 0.0 (1302 voxels, overlap=0.174)
    Left_VentralDC (28): linear fit = 1.11 x + 0.0 (1302 voxels, peak = 96), gca=96.1
    gca peak = 0.17522 (25)
    mri peak = 0.33333 (21)
    gca peak = 0.17113 (14)
    mri peak = 0.28004 (20)
    Fourth_Ventricle (15): linear fit = 1.33 x + 0.0 (117 voxels, overlap=0.283)
    Fourth_Ventricle (15): linear fit = 1.33 x + 0.0 (117 voxels, peak = 19), gca=18.6
    gca peak Unknown = 0.94777 ( 0)
    gca peak Left_Inf_Lat_Vent = 0.16627 (28)
    gca peak Third_Ventricle = 0.17522 (25)
    gca peak CSF = 0.20346 (36)
    gca peak Left_Accumbens_area = 0.70646 (62)
    gca peak Left_undetermined = 1.00000 (28)
    gca peak Left_vessel = 0.89917 (53)
    gca peak Left_choroid_plexus = 0.11689 (35)
    gca peak Right_Inf_Lat_Vent = 0.25504 (23)
    gca peak Right_Accumbens_area = 0.31650 (65)
    gca peak Right_vessel = 0.77268 (52)
    gca peak Right_choroid_plexus = 0.13275 (38)
    gca peak Fifth_Ventricle = 0.60973 (33)
    gca peak WM_hypointensities = 0.11013 (77)
    gca peak non_WM_hypointensities = 0.11354 (41)
    gca peak Optic_Chiasm = 0.51646 (76)
    not using caudate to estimate GM means
    estimating mean gm scale to be 1.17 x + 0.0
    estimating mean wm scale to be 1.00 x + 0.0
    estimating mean csf scale to be 1.08 x + 0.0
    saving intensity scales to aseg.auto_noCCseg.label_intensities.txt
    renormalizing by structure alignment....
    renormalizing input #0
    gca peak = 0.19538 (13)
    mri peak = 0.10044 (12)
    Left_Lateral_Ventricle (4): linear fit = 1.03 x + 0.0 (916 voxels, overlap=0.966)
    Left_Lateral_Ventricle (4): linear fit = 1.03 x + 0.0 (916 voxels, peak = 13), gca=13.5
    gca peak = 0.16044 (15)
    mri peak = 0.14649 (18)
    Right_Lateral_Ventricle (43): linear fit = 0.98 x + 0.0 (1514 voxels, overlap=0.597)
    Right_Lateral_Ventricle (43): linear fit = 0.98 x + 0.0 (1514 voxels, peak = 15), gca=14.6
    gca peak = 0.21930 (98)
    mri peak = 0.21672 (98)
    Right_Pallidum (52): linear fit = 1.00 x + 0.0 (646 voxels, overlap=0.957)
    Right_Pallidum (52): linear fit = 1.00 x + 0.0 (646 voxels, peak = 98), gca=98.0
    gca peak = 0.16154 (96)
    mri peak = 0.23288 (97)
    Left_Pallidum (13): linear fit = 1.01 x + 0.0 (657 voxels, overlap=0.927)
    Left_Pallidum (13): linear fit = 1.01 x + 0.0 (657 voxels, peak = 97), gca=97.4
    gca peak = 0.27314 (67)
    mri peak = 0.21287 (67)
    Right_Hippocampus (53): linear fit = 0.99 x + 0.0 (823 voxels, overlap=0.782)
    Right_Hippocampus (53): linear fit = 0.99 x + 0.0 (823 voxels, peak = 66), gca=66.0
    gca peak = 0.32959 (63)
    mri peak = 0.13483 (65)
    Left_Hippocampus (17): linear fit = 0.99 x + 0.0 (751 voxels, overlap=1.000)
    Left_Hippocampus (17): linear fit = 0.99 x + 0.0 (751 voxels, peak = 62), gca=62.1
    gca peak = 0.07884 (105)
    mri peak = 0.26841 (106)
    Right_Cerebral_White_Matter (41): linear fit = 1.00 x + 0.0 (32829 voxels, overlap=0.508)
    Right_Cerebral_White_Matter (41): linear fit = 1.00 x + 0.0 (32829 voxels, peak = 105), gca=105.0
    gca peak = 0.08137 (104)
    mri peak = 0.21465 (105)
    Left_Cerebral_White_Matter (2): linear fit = 1.00 x + 0.0 (35531 voxels, overlap=0.539)
    Left_Cerebral_White_Matter (2): linear fit = 1.00 x + 0.0 (35531 voxels, peak = 104), gca=104.0
    gca peak = 0.08972 (65)
    mri peak = 0.04782 (66)
    Left_Cerebral_Cortex (3): linear fit = 1.00 x + 0.0 (22966 voxels, overlap=0.963)
    Left_Cerebral_Cortex (3): linear fit = 1.00 x + 0.0 (22966 voxels, peak = 65), gca=65.0
    gca peak = 0.10309 (64)
    mri peak = 0.05009 (63)
    Right_Cerebral_Cortex (42): linear fit = 1.02 x + 0.0 (20600 voxels, overlap=0.944)
    Right_Cerebral_Cortex (42): linear fit = 1.02 x + 0.0 (20600 voxels, peak = 66), gca=65.6
    gca peak = 0.22735 (75)
    mri peak = 0.13764 (78)
    Right_Caudate (50): linear fit = 1.00 x + 0.0 (819 voxels, overlap=1.000)
    Right_Caudate (50): linear fit = 1.00 x + 0.0 (819 voxels, peak = 75), gca=75.0
    gca peak = 0.14473 (69)
    mri peak = 0.16890 (77)
    Left_Caudate (11): linear fit = 1.00 x + 0.0 (821 voxels, overlap=0.676)
    Left_Caudate (11): linear fit = 1.00 x + 0.0 (821 voxels, peak = 69), gca=69.0
    gca peak = 0.12064 (65)
    mri peak = 0.07530 (66)
    Left_Cerebellum_Cortex (8): linear fit = 1.01 x + 0.0 (22259 voxels, overlap=0.949)
    Left_Cerebellum_Cortex (8): linear fit = 1.01 x + 0.0 (22259 voxels, peak = 66), gca=66.0
    gca peak = 0.12644 (67)
    mri peak = 0.07220 (68)
    Right_Cerebellum_Cortex (47): linear fit = 0.99 x + 0.0 (23168 voxels, overlap=0.976)
    Right_Cerebellum_Cortex (47): linear fit = 0.99 x + 0.0 (23168 voxels, peak = 66), gca=66.0
    gca peak = 0.15939 (91)
    mri peak = 0.25949 (91)
    Left_Cerebellum_White_Matter (7): linear fit = 1.00 x + 0.0 (5523 voxels, overlap=0.746)
    Left_Cerebellum_White_Matter (7): linear fit = 1.00 x + 0.0 (5523 voxels, peak = 91), gca=91.0
    gca peak = 0.14957 (92)
    mri peak = 0.17737 (90)
    Right_Cerebellum_White_Matter (46): linear fit = 0.99 x + 0.0 (5334 voxels, overlap=0.791)
    Right_Cerebellum_White_Matter (46): linear fit = 0.99 x + 0.0 (5334 voxels, peak = 91), gca=90.6
    gca peak = 0.22741 (72)
    mri peak = 0.16252 (72)
    Left_Amygdala (18): linear fit = 1.00 x + 0.0 (378 voxels, overlap=0.970)
    Left_Amygdala (18): linear fit = 1.00 x + 0.0 (378 voxels, peak = 72), gca=72.0
    gca peak = 0.24938 (70)
    mri peak = 0.14222 (70)
    Right_Amygdala (54): linear fit = 1.00 x + 0.0 (450 voxels, overlap=1.002)
    Right_Amygdala (54): linear fit = 1.00 x + 0.0 (450 voxels, peak = 70), gca=70.0
    gca peak = 0.09894 (93)
    mri peak = 0.08966 (92)
    Left_Thalamus (10): linear fit = 1.00 x + 0.0 (4239 voxels, overlap=0.944)
    Left_Thalamus (10): linear fit = 1.00 x + 0.0 (4239 voxels, peak = 93), gca=92.5
    gca peak = 0.10165 (89)
    mri peak = 0.09267 (94)
    Right_Thalamus (49): linear fit = 1.00 x + 0.0 (3616 voxels, overlap=0.934)
    Right_Thalamus (49): linear fit = 1.00 x + 0.0 (3616 voxels, peak = 89), gca=88.6
    gca peak = 0.08071 (86)
    mri peak = 0.13502 (83)
    Left_Putamen (12): linear fit = 1.00 x + 0.0 (1573 voxels, overlap=0.728)
    Left_Putamen (12): linear fit = 1.00 x + 0.0 (1573 voxels, peak = 86), gca=86.0
    gca peak = 0.08515 (83)
    mri peak = 0.11726 (83)
    Right_Putamen (51): linear fit = 0.98 x + 0.0 (1825 voxels, overlap=0.899)
    Right_Putamen (51): linear fit = 0.98 x + 0.0 (1825 voxels, peak = 81), gca=80.9
    gca peak = 0.08056 (89)
    mri peak = 0.10605 (90)
    Brain_Stem (16): linear fit = 0.98 x + 0.0 (10876 voxels, overlap=0.681)
    Brain_Stem (16): linear fit = 0.98 x + 0.0 (10876 voxels, peak = 87), gca=86.8
    gca peak = 0.11663 (97)
    mri peak = 0.13885 (100)
    Right_VentralDC (60): linear fit = 1.00 x + 0.0 (1312 voxels, overlap=0.746)
    Right_VentralDC (60): linear fit = 1.00 x + 0.0 (1312 voxels, peak = 97), gca=97.5
    gca peak = 0.14874 (96)
    mri peak = 0.16600 (100)
    Left_VentralDC (28): linear fit = 1.00 x + 0.0 (1302 voxels, overlap=0.803)
    Left_VentralDC (28): linear fit = 1.00 x + 0.0 (1302 voxels, peak = 96), gca=95.5
    gca peak = 0.16499 (28)
    mri peak = 0.33333 (21)
    gca peak = 0.16433 (21)
    mri peak = 0.28004 (20)
    Fourth_Ventricle (15): linear fit = 0.99 x + 0.0 (117 voxels, overlap=0.741)
    Fourth_Ventricle (15): linear fit = 0.99 x + 0.0 (117 voxels, peak = 21), gca=20.7
    gca peak Unknown = 0.94777 ( 0)
    gca peak Left_Inf_Lat_Vent = 0.19147 (31)
    gca peak Third_Ventricle = 0.16499 (28)
    gca peak CSF = 0.18962 (39)
    gca peak Left_Accumbens_area = 0.54460 (62)
    gca peak Left_undetermined = 1.00000 (28)
    gca peak Left_vessel = 0.89917 (53)
    gca peak Left_choroid_plexus = 0.11541 (35)
    gca peak Right_Inf_Lat_Vent = 0.26261 (28)
    gca peak Right_Accumbens_area = 0.30638 (73)
    gca peak Right_vessel = 0.77268 (52)
    gca peak Right_choroid_plexus = 0.13275 (38)
    gca peak Fifth_Ventricle = 0.73548 (35)
    gca peak WM_hypointensities = 0.11093 (77)
    gca peak non_WM_hypointensities = 0.14635 (41)
    gca peak Optic_Chiasm = 0.51659 (76)
    not using caudate to estimate GM means
    estimating mean gm scale to be 1.00 x + 0.0
    estimating mean wm scale to be 1.00 x + 0.0
    estimating mean csf scale to be 1.00 x + 0.0
    saving intensity scales to aseg.auto_noCCseg.label_intensities.txt
    saving sequentially combined intensity scales to aseg.auto_noCCseg.label_intensities.txt
    33606 voxels changed in iteration 0 of unlikely voxel relabeling
    154 voxels changed in iteration 1 of unlikely voxel relabeling
    5 voxels changed in iteration 2 of unlikely voxel relabeling
    0 voxels changed in iteration 3 of unlikely voxel relabeling
    28971 gm and wm labels changed (%38 to gray, %62 to white out of all changed labels)
    457 hippocampal voxels changed.
    0 amygdala voxels changed.
    Reclassifying using Gibbs Priors
    pass 1: 47706 changed. image ll: -2.036, PF=0.500
    pass 2: 11160 changed. image ll: -2.036, PF=0.500
    pass 3: 3295 changed.
    23625 voxels changed in iteration 0 of unlikely voxel relabeling
    220 voxels changed in iteration 1 of unlikely voxel relabeling
    18 voxels changed in iteration 2 of unlikely voxel relabeling
    0 voxels changed in iteration 3 of unlikely voxel relabeling
    5364 voxels changed in iteration 0 of unlikely voxel relabeling
    137 voxels changed in iteration 1 of unlikely voxel relabeling
    9 voxels changed in iteration 2 of unlikely voxel relabeling
    1 voxels changed in iteration 3 of unlikely voxel relabeling
    0 voxels changed in iteration 4 of unlikely voxel relabeling
    4516 voxels changed in iteration 0 of unlikely voxel relabeling
    83 voxels changed in iteration 1 of unlikely voxel relabeling
    0 voxels changed in iteration 2 of unlikely voxel relabeling
    4023 voxels changed in iteration 0 of unlikely voxel relabeling
    26 voxels changed in iteration 1 of unlikely voxel relabeling
    2 voxels changed in iteration 2 of unlikely voxel relabeling
    1 voxels changed in iteration 3 of unlikely voxel relabeling
    0 voxels changed in iteration 4 of unlikely voxel relabeling
     !!!!!!!!! ventricle segment 0 with volume 5074 above threshold 100 - not erasing !!!!!!!!!!
     !!!!!!!!! ventricle segment 1 with volume 6332 above threshold 100 - not erasing !!!!!!!!!!
    writing labeled volume to aseg.auto_noCCseg.mgz
    mri_ca_label utimesec    2357.275314
    mri_ca_label stimesec    15.937804
    mri_ca_label ru_maxrss   2129660
    mri_ca_label ru_ixrss    0
    mri_ca_label ru_idrss    0
    mri_ca_label ru_isrss    0
    mri_ca_label ru_minflt   11280482
    mri_ca_label ru_majflt   0
    mri_ca_label ru_nswap    0
    mri_ca_label ru_inblock  0
    mri_ca_label ru_oublock  608
    mri_ca_label ru_msgsnd   0
    mri_ca_label ru_msgrcv   0
    mri_ca_label ru_nsignals 0
    mri_ca_label ru_nvcsw    635
    mri_ca_label ru_nivcsw   5726
    auto-labeling took 35 minutes and 7 seconds.
    #--------------------------------------
    #@# CC Seg Mon Feb  5 03:14:39 UTC 2024

     mri_cc -aseg aseg.auto_noCCseg.mgz -o aseg.auto.mgz -lta /out/sourcedata/freesurfer/sub-CH8698b/mri/transforms/cc_up.lta sub-CH8698b 

    will read input aseg from aseg.auto_noCCseg.mgz
    writing aseg with cc labels to aseg.auto.mgz
    will write lta as /out/sourcedata/freesurfer/sub-CH8698b/mri/transforms/cc_up.lta
    reading aseg from /out/sourcedata/freesurfer/sub-CH8698b/mri/aseg.auto_noCCseg.mgz
    reading norm from /out/sourcedata/freesurfer/sub-CH8698b/mri/norm.mgz
    29359 voxels in left wm, 30623 in right wm, xrange [125, 132]
    searching rotation angles z=[-6  8], y=[-8  6]

    searching scale 1 Z rot -6.0  
    searching scale 1 Z rot -5.8  
    searching scale 1 Z rot -5.5  
    searching scale 1 Z rot -5.3  
    searching scale 1 Z rot -5.0  
    searching scale 1 Z rot -4.8  
    searching scale 1 Z rot -4.5  
    searching scale 1 Z rot -4.3  
    searching scale 1 Z rot -4.0  
    searching scale 1 Z rot -3.8  
    searching scale 1 Z rot -3.5  
    searching scale 1 Z rot -3.3  
    searching scale 1 Z rot -3.0  
    searching scale 1 Z rot -2.8  
    searching scale 1 Z rot -2.5  
    searching scale 1 Z rot -2.3  
    searching scale 1 Z rot -2.0  
    searching scale 1 Z rot -1.8  
    searching scale 1 Z rot -1.5  
    searching scale 1 Z rot -1.3  
    searching scale 1 Z rot -1.0  
    searching scale 1 Z rot -0.8  
    searching scale 1 Z rot -0.5  
    searching scale 1 Z rot -0.3  
    searching scale 1 Z rot -0.0  
    searching scale 1 Z rot 0.2  
    searching scale 1 Z rot 0.5  
    searching scale 1 Z rot 0.7  
    searching scale 1 Z rot 1.0  
    searching scale 1 Z rot 1.2  
    searching scale 1 Z rot 1.5  
    searching scale 1 Z rot 1.7  
    searching scale 1 Z rot 2.0  
    searching scale 1 Z rot 2.2  
    searching scale 1 Z rot 2.5  
    searching scale 1 Z rot 2.7  
    searching scale 1 Z rot 3.0  
    searching scale 1 Z rot 3.2  
    searching scale 1 Z rot 3.5  
    searching scale 1 Z rot 3.7  
    searching scale 1 Z rot 4.0  
    searching scale 1 Z rot 4.2  
    searching scale 1 Z rot 4.5  
    searching scale 1 Z rot 4.7  
    searching scale 1 Z rot 5.0  
    searching scale 1 Z rot 5.2  
    searching scale 1 Z rot 5.5  
    searching scale 1 Z rot 5.7  
    searching scale 1 Z rot 6.0  
    searching scale 1 Z rot 6.2  
    searching scale 1 Z rot 6.5  
    searching scale 1 Z rot 6.7  
    searching scale 1 Z rot 7.0  
    searching scale 1 Z rot 7.2  
    searching scale 1 Z rot 7.5  
    searching scale 1 Z rot 7.7  global minimum found at slice 128.0, rotations (-1.12, 0.99)
    final transformation (x=128.0, yr=-1.118, zr=0.994):
     0.99966  -0.01734  -0.01951   3.63136;
     0.01734   0.99985  -0.00034   43.83077;
     0.01952   0.00000   0.99981   14.52321;
     0.00000   0.00000   0.00000   1.00000;
    updating x range to be [126, 131] in xformed coordinates
    best xformed slice 128
    min_x_fornix = 132
    min_x_fornix = 135
    WARNING: min_x_fornix not set
    min_x_fornix = 139
    min_x_fornix = 139
    cc center is found at 128 82 111
    eigenvectors:
    -0.00040   0.00302   1.00000;
     0.15001  -0.98868   0.00305;
     0.98868   0.15001  -0.00005;
    writing aseg with callosum to /out/sourcedata/freesurfer/sub-CH8698b/mri/aseg.auto.mgz...
    corpus callosum segmentation took 0.6 minutes
    #VMPC# mri_cc VmPeak  947804
    mri_cc done
    #--------------------------------------
    #@# Merge ASeg Mon Feb  5 03:15:15 UTC 2024

     cp aseg.auto.mgz aseg.presurf.mgz 

    #--------------------------------------------
    #@# Intensity Normalization2 Mon Feb  5 03:15:15 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8698b/mri

     mri_normalize -seed 1234 -mprage -aseg aseg.presurf.mgz -mask brainmask.mgz norm.mgz brain.mgz 

    setting seed for random number genererator to 1234
    assuming input volume is MGH (Van der Kouwe) MP-RAGE
    using segmentation for initial intensity normalization
    using MR volume brainmask.mgz to mask input volume...
    reading mri_src from norm.mgz...
    Reading aseg aseg.presurf.mgz
    normalizing image...
    NOT doing gentle normalization with control points/label
    processing with aseg
    removing outliers in the aseg WM...
    326 control points removed
    Building bias image
    building Voronoi diagram...
    performing soap bubble smoothing, sigma = 0...
    Smoothing with sigma 8
    Applying bias correction
    building Voronoi diagram...
    performing soap bubble smoothing, sigma = 8...

    Iterating 2 times
    ---------------------------------
    3d normalization pass 1 of 2
    white matter peak found at 110
    white matter peak found at 110
    gm peak at 72 (72), valley at 59 (59)
    csf peak at 37, setting threshold to 60
    building Voronoi diagram...
    performing soap bubble smoothing, sigma = 8...
    ---------------------------------
    3d normalization pass 2 of 2
    white matter peak found at 110
    white matter peak found at 110
    gm peak at 74 (74), valley at 33 (33)
    csf peak at 38, setting threshold to 62
    building Voronoi diagram...
    performing soap bubble smoothing, sigma = 8...
    Done iterating ---------------------------------
    writing output to brain.mgz
    3D bias adjustment took 2 minutes and 33 seconds.
    #--------------------------------------------
    #@# Mask BFS Mon Feb  5 03:17:50 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8698b/mri

     mri_mask -T 5 brain.mgz brainmask.mgz brain.finalsurfs.mgz 

    threshold mask volume at 5
    DoAbs = 0
    Found 1219769 voxels in mask (pct=  7.27)
    Writing masked volume to brain.finalsurfs.mgz...done.
    #--------------------------------------------
    #@# WM Segmentation Mon Feb  5 03:17:51 UTC 2024

     AntsDenoiseImageFs -i brain.mgz -o antsdn.brain.mgz 

     mri_segment -wsizemm 13 -mprage antsdn.brain.mgz wm.seg.mgz 

    wsizemm = 13, voxres = 1, wsize = 13
    Widening wm low from 89 to 79
    assuming input volume is MGH (Van der Kouwe) MP-RAGE
    wm mean:  110
    wsize:    13
    wm low:   79
    wm hi:    125
    gray low: 30
    gray hi:  99
    Doing initial trinary intensity segmentation 
    Using local statistics to label ambiguous voxels
    Autodetecting stats
    Computing class statistics for intensity windows...
    CCS WM (102.0): 101.4 +- 6.4 [79.0 --> 118.0]
    CCS GM (76.0) : 75.2 +- 9.4 [30.0 --> 95.0]
     white_mean 101.448
     white_sigma 6.42697
     gray_mean 75.1583
     gray_sigma 9.41918
    setting bottom of white matter range wm_low to 84.6
    setting top of gray matter range gray_hi to 94.0
     wm_low 84.5775
     wm_hi  125
     gray_low 30
     gray_hi  93.9967
    Redoing initial intensity segmentation...
    Recomputing local statistics to label ambiguous voxels...
     wm_low 84.5775
     wm_hi  125
     gray_low 30
     gray_hi  93.9967
    using local geometry to label remaining ambiguous voxels...
    polvwsize = 5, polvlen = 3, gray_hi = 93.9967, wm_low = 84.5775
    MRIcpolvMedianCurveSegment(): wsize=5, len=3, gmhi=93.9967, wmlow=84.5775
        103321 voxels processed (0.62%)
         43542 voxels white (0.26%)
         59779 voxels non-white (0.36%)

    Reclassifying voxels using Gaussian border classifier niter=1
    MRIreclassify(): wm_low=79.5775, gray_hi=93.9967, wsize=13
        204895 voxels tested (1.22%)
         37349 voxels changed (0.22%)
         48274 multi-scale searches  (0.29%)
    Recovering bright white
    MRIrecoverBrightWhite()
     wm_low 84.5775
     wm_hi 125
     slack 6.42697
     pct_thresh 0.33
     intensity_thresh 131.427
     nvox_thresh 8.58
            0 voxels tested (0.00%)
            0 voxels changed (0.00%)

    removing voxels with positive offset direction...
    MRIremoveWrongDirection() wsize=3, lowthr=79.5775, hithr=93.9967
      smoothing input volume with sigma = 0.250
        62843 voxels tested (0.37%)
        12460 voxels changed (0.07%)
    thicken = 1
    removing 1-dimensional structures...
    MRIremove1dStructures(): max_iter=10000, thresh=2, WM_MIN_VAL=5
     1194 sparsely connected voxels removed in 1 iterations
    thickening thin strands....
    thickness 4
    nsegments 20
    wm_hi 125
    767 diagonally connected voxels added...
    MRIthickenThinWMStrands(): thickness=4, nsegments=20
      20 segments, 4179 filled
    MRIfindBrightNonWM(): 0 bright non-wm voxels segmented.
    MRIfilterMorphology() WM_MIN_VAL=5, DIAGONAL_FILL=230
    white matter segmentation took 1.3 minutes
    writing output to wm.seg.mgz...

     mri_edit_wm_with_aseg -keep-in wm.seg.mgz brain.mgz aseg.presurf.mgz wm.asegedit.mgz 

    preserving editing changes in input volume...
    auto filling took 0.55 minutes
    reading wm segmentation from wm.seg.mgz...
    0 voxels added to wm to prevent paths from MTL structures to cortex
    2743 additional wm voxels added
    0 additional wm voxels added
    SEG EDIT: 35218 voxels turned on, 36481 voxels turned off.
    propagating editing to output volume from wm.seg.mgz
    writing edited volume to wm.asegedit.mgz....

     mri_pretess wm.asegedit.mgz wm norm.mgz wm.mgz 

    Iteration Number : 1
    pass   1 (xy+):   4 found -   4 modified     |    TOTAL:   4
    pass   2 (xy+):   0 found -   4 modified     |    TOTAL:   4
    pass   1 (xy-):  11 found -  11 modified     |    TOTAL:  15
    pass   2 (xy-):   0 found -  11 modified     |    TOTAL:  15
    pass   1 (yz+):  20 found -  20 modified     |    TOTAL:  35
    pass   2 (yz+):   0 found -  20 modified     |    TOTAL:  35
    pass   1 (yz-):  11 found -  11 modified     |    TOTAL:  46
    pass   2 (yz-):   0 found -  11 modified     |    TOTAL:  46
    pass   1 (xz+):   7 found -   7 modified     |    TOTAL:  53
    pass   2 (xz+):   0 found -   7 modified     |    TOTAL:  53
    pass   1 (xz-):   9 found -   9 modified     |    TOTAL:  62
    pass   2 (xz-):   0 found -   9 modified     |    TOTAL:  62
    Iteration Number : 1
    pass   1 (+++):   4 found -   4 modified     |    TOTAL:   4
    pass   2 (+++):   0 found -   4 modified     |    TOTAL:   4
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   4
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   4
    pass   1 (+++):   2 found -   2 modified     |    TOTAL:   6
    pass   2 (+++):   0 found -   2 modified     |    TOTAL:   6
    Iteration Number : 1
    pass   1 (++):  21 found -  21 modified     |    TOTAL:  21
    pass   2 (++):   0 found -  21 modified     |    TOTAL:  21
    pass   1 (+-):  35 found -  35 modified     |    TOTAL:  56
    pass   2 (+-):   0 found -  35 modified     |    TOTAL:  56
    pass   1 (--):  13 found -  13 modified     |    TOTAL:  69
    pass   2 (--):   0 found -  13 modified     |    TOTAL:  69
    pass   1 (-+):  43 found -  43 modified     |    TOTAL: 112
    pass   2 (-+):   0 found -  43 modified     |    TOTAL: 112
    Iteration Number : 2
    pass   1 (xy+):   1 found -   1 modified     |    TOTAL:   1
    pass   2 (xy+):   0 found -   1 modified     |    TOTAL:   1
    pass   1 (xy-):   1 found -   1 modified     |    TOTAL:   2
    pass   2 (xy-):   0 found -   1 modified     |    TOTAL:   2
    pass   1 (yz+):   0 found -   0 modified     |    TOTAL:   2
    pass   1 (yz-):   0 found -   0 modified     |    TOTAL:   2
    pass   1 (xz+):   1 found -   1 modified     |    TOTAL:   3
    pass   2 (xz+):   0 found -   1 modified     |    TOTAL:   3
    pass   1 (xz-):   0 found -   0 modified     |    TOTAL:   3
    Iteration Number : 2
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 2
    pass   1 (++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (--):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (-+):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 3
    pass   1 (xy+):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (xy-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (yz+):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (yz-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (xz+):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (xz-):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 3
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+++):   0 found -   0 modified     |    TOTAL:   0
    Iteration Number : 3
    pass   1 (++):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (+-):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (--):   0 found -   0 modified     |    TOTAL:   0
    pass   1 (-+):   0 found -   0 modified     |    TOTAL:   0

    Total Number of Modified Voxels = 183 (out of 465866: 0.039282)
    binarizing input wm segmentation...
    Ambiguous edge configurations... 

    mri_pretess done

    #--------------------------------------------
    #@# Fill Mon Feb  5 03:20:12 UTC 2024
    /out/sourcedata/freesurfer/sub-CH8698b/mri

     mri_fill -a ../scripts/ponscc.cut.log -xform transforms/talairach.lta -segmentation aseg.presurf.mgz -ctab /opt/freesurfer/SubCorticalMassLUT.txt wm.mgz filled.mgz 

    logging cutting plane coordinates to ../scripts/ponscc.cut.log...
    INFO: Using transforms/talairach.lta and its offset for Talairach volume ...
    using segmentation aseg.presurf.mgz...
    reading input volume...done.
    searching for cutting planes...voxel to talairach voxel transform
     1.16735  -0.03830  -0.00903  -19.57378;
     0.04113   1.23200   0.05566  -14.05557;
     0.00973  -0.05271   1.03939  -3.43275;
     0.00000   0.00000   0.00000   1.00000;
    voxel to talairach voxel transform
     1.16735  -0.03830  -0.00903  -19.57378;
     0.04113   1.23200   0.05566  -14.05557;
     0.00973  -0.05271   1.03939  -3.43275;
     0.00000   0.00000   0.00000   1.00000;
    reading segmented volume aseg.presurf.mgz
    removing CC from segmentation
    Looking for area (min, max) = (350, 1400)
    area[0] = 2369 (min = 350, max = 1400), aspect = 1.32 (min = 0.10, max = 0.75)
    need search nearby
    using seed (126, 123, 151), TAL = (2.0, 23.0, 5.0)
    talairach voxel to voxel transform
     0.85565   0.02685   0.00599   17.14623;
    -0.02814   0.80895  -0.04357   10.66991;
    -0.00943   0.04077   0.95983   3.68335;
     0.00000   0.00000   0.00000   1.00000;
    segmentation indicates cc at (126,  123,  151) --> (2.0, 23.0, 5.0)
    done.
    filling took 1.0 minutes
    talairach cc position changed to (2.00, 23.00, 5.00)
    Erasing brainstem...done.
    seed_search_size = 9, min_neighbors = 5
    search rh wm seed point around talairach space:(20.00, 23.00, 5.00) SRC: (113.76, 100.55, 152.61)
    search lh wm seed point around talairach space (-16.00, 23.00, 5.00), SRC: (144.57, 99.54, 152.28)
    compute mri_fill using aseg
    Erasing Brain Stem and Cerebellum ...
    Define left and right masks using aseg:
    Building Voronoi diagram ...
    Using the Voronoi diagram for separating WM into two hemispheres ...
    Find the largest connected component for each hemisphere ...
    Embedding colortable
    mri_fill done, writing output to filled.mgz...
     cp filled.mgz filled.auto.mgz
    cp: skipping file 'filled.mgz', as it was replaced while being copied
    Linux a6080e14df56 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

    recon-all -s sub-CH8698b exited with ERRORS at Mon Feb  5 03:21:14 UTC 2024

    For more details, see the log file /out/sourcedata/freesurfer/sub-CH8698b/scripts/recon-all.log
    To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Stderr:
    cat: standard output: Stale file handle
    cat: standard output: Stale file handle
Traceback:
    RuntimeError: subprocess exited with code 1.
effigies commented 7 months ago

Copy or symlink failures make me think it's probably a filesystem capabilities issue. What type of filesystem is /Volumes/NMorphCHderivatives? If it's a FAT/NTFS filesystem, it's possible that you're hitting some issues with a non-POSIX FS appearing to the process as POSIX.

isacdaavid commented 7 months ago

Your suggestion is making perfect sense, the data lives in a windows/cifs/samba NAS.

I have taken a copy out to the local filesystem (ext4), and so far fmriprep has perfectly preprocessed 4 subjects without errors. I will come back tomorrow with a higher sample, hopefully just to close the bug report.

Thank you.