still getting permission denied errors with freesurfer files

[jbwexler]

We had hoped that by unlocking freesurfer/*/scripts we would avoid permission denied errors when rerunning freesurfer/fmriprep. It worked for some datasets but not for all. @effigies thinks this may be caused by the same issue causing https://github.com/poldracklab/tacc-openneuro/issues/1. As an example, the files that caused permission denied errors in ds000171 were aseg.auto_noCCseg.label_intensities.txt and talairach.label_intensities.txt.

Here's a crash file:

(main) login4.frontera(1017)$ cat /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sub-mdd11/log/20220301-135053_8210f5ce-0568-40bf-abb4-25f2df5cfbf7/crash-20220301-135904-jbwexler-autorecon2_vol-07f2a82d-6964-4450-b4c7-31971bc0de28.txt Node: fmriprep_wf.single_subject_mdd11_wf.anat_preproc_wf.surface_recon_wf.autorecon_resume_wf.autorecon2_vol Working directory: /scratch1/03201/jbwexler/work_dir/fmriprep/ds000171_sub-mdd11/fmriprep_wf/single_subject_mdd11_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon2_vol

Node inputs:

FLAIR_file = T1_files = T2_file = args = big_ventricles = brainstem = directive = autorecon2-volonly environ = {} expert = flags = hemi = hippocampal_subfields_T1 = hippocampal_subfields_T2 = hires = mprage = mri_aparc2aseg = mri_ca_label = mri_ca_normalize = mri_ca_register = mri_edit_wm_with_aseg = mri_em_register = mri_fill = mri_mask = mri_normalize = mri_pretess = mri_remove_neck = mri_segment = mri_segstats = mri_tessellate = mri_watershed = mris_anatomical_stats = mris_ca_label = mris_fix_topology = mris_inflate = mris_make_surfaces = mris_register = mris_smooth = mris_sphere = mris_surf2vol = mrisp_paint = openmp = 7 parallel = steps = subject_id = recon_all subjects_dir = talairach = use_FLAIR = use_T2 = xopts =

Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/opt/conda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 428, in run runtime = self._run_interface(runtime) File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 822, in _run_interface self.raise_exception(runtime) File "/opt/conda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 749, in raise_exception raise RuntimeError( RuntimeError: Command: recon-all -autorecon2-volonly -openmp 7 -subjid sub-mdd11 -sd /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer -nogcareg -nocanorm -nocareg Standard output: Subject Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a Current Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a INFO: SUBJECTS_DIR is /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer Actual FREESURFER_HOME /opt/freesurfer -rw-r--r--+ 1 jbwexler G-802037 133469 Feb 18 17:18 /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer/sub-mdd11/scripts/recon-all.log Linux c187-024.frontera.tacc.utexas.edu 3.10.0-1160.45.1.el7.x86_64 #1 SMP Wed Oct 13 17:20:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux '/opt/freesurfer/bin/recon-all' -> '/scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer/sub-mdd11/scripts/recon-all.local-copy'

--------------------------------------

@# SubCort Seg Tue Mar 1 13:55:24 UTC 2022

mri_ca_label -relabel_unlikely 9 .3 -prior 0.5 -align norm.mgz transforms/talairach.m3z /opt/freesurfer/average/RB_all_2016-05-10.vc700.gca aseg.auto_noCCseg.mgz

sysname Linux hostname c187-024.frontera.tacc.utexas.edu machine x86_64

setenv SUBJECTS_DIR /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer cd /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer/sub-mdd11/mri mri_ca_label -relabel_unlikely 9 .3 -prior 0.5 -align norm.mgz transforms/talairach.m3z /opt/freesurfer/average/RB_all_2016-05-10.vc700.gca aseg.auto_noCCseg.mgz

== Number of threads available to mri_ca_label for OpenMP = 7 == relabeling unlikely voxels with window_size = 9 and prior threshold 0.30 using Gibbs prior factor = 0.500 renormalizing sequences with structure alignment, equivalent to: -renormalize -renormalize_mean 0.500 -regularize 0.500 reading 1 input volumes reading classifier array from /opt/freesurfer/average/RB_all_2016-05-10.vc700.gca reading input volume from norm.mgz average std[0] = 7.3 reading transform from transforms/talairach.m3z setting orig areas to linear transform determinant scaled 6.42 Atlas used for the 3D morph was /opt/freesurfer/average/RB_all_2016-05-10.vc700.gca average std = 7.3 using min determinant for regularization = 5.3 0 singular and 0 ill-conditioned covariance matrices regularized labeling volume... renormalizing by structure alignment.... renormalizing input #0 gca peak = 0.16259 (20) mri peak = 0.12587 (19) Left_Lateral_Ventricle (4): linear fit = 1.02 x + 0.0 (796 voxels, overlap=0.907) Left_Lateral_Ventricle (4): linear fit = 1.02 x + 0.0 (796 voxels, peak = 20), gca=20.5 gca peak = 0.17677 (13) mri peak = 0.13066 (22) Right_Lateral_Ventricle (43): linear fit = 1.47 x + 0.0 (802 voxels, overlap=0.502) Right_Lateral_Ventricle (43): linear fit = 1.47 x + 0.0 (802 voxels, peak = 19), gca=19.0 gca peak = 0.28129 (95) mri peak = 0.10622 (103) Right_Pallidum (52): linear fit = 1.08 x + 0.0 (659 voxels, overlap=0.668) Right_Pallidum (52): linear fit = 1.08 x + 0.0 (659 voxels, peak = 102), gca=102.1 gca peak = 0.16930 (96) mri peak = 0.08889 (105) Left_Pallidum (13): linear fit = 1.10 x + 0.0 (647 voxels, overlap=0.758) Left_Pallidum (13): linear fit = 1.10 x + 0.0 (647 voxels, peak = 105), gca=105.1 gca peak = 0.24553 (55) mri peak = 0.08611 (64) Right_Hippocampus (53): linear fit = 1.13 x + 0.0 (1026 voxels, overlap=0.481) Right_Hippocampus (53): linear fit = 1.13 x + 0.0 (1026 voxels, peak = 62), gca=62.4 gca peak = 0.30264 (59) mri peak = 0.09124 (70) Left_Hippocampus (17): linear fit = 1.14 x + 0.0 (830 voxels, overlap=0.133) Left_Hippocampus (17): linear fit = 1.14 x + 0.0 (830 voxels, peak = 68), gca=67.6 gca peak = 0.07580 (103) mri peak = 0.12617 (106) Right_Cerebral_White_Matter (41): linear fit = 1.03 x + 0.0 (42695 voxels, overlap=0.548) Right_Cerebral_White_Matter (41): linear fit = 1.03 x + 0.0 (42695 voxels, peak = 107), gca=106.6 gca peak = 0.07714 (104) mri peak = 0.12086 (108) Left_Cerebral_White_Matter (2): linear fit = 1.03 x + 0.0 (44992 voxels, overlap=0.465) Left_Cerebral_White_Matter (2): linear fit = 1.03 x + 0.0 (44992 voxels, peak = 108), gca=107.6 gca peak = 0.09712 (58) mri peak = 0.04226 (61) Left_Cerebral_Cortex (3): linear fit = 1.08 x + 0.0 (38483 voxels, overlap=0.860) Left_Cerebral_Cortex (3): linear fit = 1.08 x + 0.0 (38483 voxels, peak = 62), gca=62.4 gca peak = 0.11620 (58) mri peak = 0.04293 (62) Right_Cerebral_Cortex (42): linear fit = 1.10 x + 0.0 (37807 voxels, overlap=0.806) Right_Cerebral_Cortex (42): linear fit = 1.10 x + 0.0 (37807 voxels, peak = 64), gca=63.5 gca peak = 0.30970 (66) mri peak = 0.13886 (77) Right_Caudate (50): linear fit = 1.14 x + 0.0 (1037 voxels, overlap=0.013) Right_Caudate (50): linear fit = 1.14 x + 0.0 (1037 voxels, peak = 76), gca=75.6 gca peak = 0.15280 (69) mri peak = 0.14414 (78) Left_Caudate (11): linear fit = 1.02 x + 0.0 (1053 voxels, overlap=0.673) Left_Caudate (11): linear fit = 1.02 x + 0.0 (1053 voxels, peak = 71), gca=70.7 gca peak = 0.13902 (56) mri peak = 0.05975 (67) Left_Cerebellum_Cortex (8): linear fit = 1.15 x + 0.0 (26471 voxels, overlap=0.331) Left_Cerebellum_Cortex (8): linear fit = 1.15 x + 0.0 (26471 voxels, peak = 65), gca=64.7 gca peak = 0.14777 (55) mri peak = 0.06168 (67) Right_Cerebellum_Cortex (47): linear fit = 1.17 x + 0.0 (27560 voxels, overlap=0.321) Right_Cerebellum_Cortex (47): linear fit = 1.17 x + 0.0 (27560 voxels, peak = 65), gca=64.6 gca peak = 0.16765 (84) mri peak = 0.11489 (93) Left_Cerebellum_White_Matter (7): linear fit = 1.09 x + 0.0 (6246 voxels, overlap=0.530) Left_Cerebellum_White_Matter (7): linear fit = 1.09 x + 0.0 (6246 voxels, peak = 91), gca=91.1 gca peak = 0.18739 (84) mri peak = 0.11928 (91) Right_Cerebellum_White_Matter (46): linear fit = 1.09 x + 0.0 (6363 voxels, overlap=0.466) Right_Cerebellum_White_Matter (46): linear fit = 1.09 x + 0.0 (6363 voxels, peak = 91), gca=91.1 gca peak = 0.29869 (57) mri peak = 0.09561 (67) Left_Amygdala (18): linear fit = 1.12 x + 0.0 (508 voxels, overlap=0.057) Left_Amygdala (18): linear fit = 1.12 x + 0.0 (508 voxels, peak = 64), gca=63.6 gca peak = 0.33601 (57) mri peak = 0.10009 (65) Right_Amygdala (54): linear fit = 1.13 x + 0.0 (537 voxels, overlap=0.296) Right_Amygdala (54): linear fit = 1.13 x + 0.0 (537 voxels, peak = 65), gca=64.7 gca peak = 0.11131 (90) mri peak = 0.07181 (91) Left_Thalamus_Proper (10): linear fit = 1.03 x + 0.0 (4342 voxels, overlap=0.877) Left_Thalamus_Proper (10): linear fit = 1.03 x + 0.0 (4342 voxels, peak = 93), gca=93.1 gca peak = 0.11793 (83) mri peak = 0.08579 (88) Right_Thalamus_Proper (49): linear fit = 1.04 x + 0.0 (4270 voxels, overlap=0.863) Right_Thalamus_Proper (49): linear fit = 1.04 x + 0.0 (4270 voxels, peak = 87), gca=86.7 gca peak = 0.08324 (81) mri peak = 0.09452 (84) Left_Putamen (12): linear fit = 1.07 x + 0.0 (2055 voxels, overlap=0.706) Left_Putamen (12): linear fit = 1.07 x + 0.0 (2055 voxels, peak = 86), gca=86.3 gca peak = 0.10360 (77) mri peak = 0.07613 (85) Right_Putamen (51): linear fit = 1.08 x + 0.0 (2265 voxels, overlap=0.812) Right_Putamen (51): linear fit = 1.08 x + 0.0 (2265 voxels, peak = 83), gca=82.8 gca peak = 0.08424 (78) mri peak = 0.09866 (88) Brain_Stem (16): linear fit = 1.08 x + 0.0 (12276 voxels, overlap=0.476) Brain_Stem (16): linear fit = 1.08 x + 0.0 (12276 voxels, peak = 84), gca=83.9 gca peak = 0.12631 (89) mri peak = 0.08556 (93) Right_VentralDC (60): linear fit = 1.10 x + 0.0 (1610 voxels, overlap=0.490) Right_VentralDC (60): linear fit = 1.10 x + 0.0 (1610 voxels, peak = 97), gca=97.5 gca peak = 0.14500 (87) mri peak = 0.09462 (93) Left_VentralDC (28): linear fit = 1.07 x + 0.0 (1713 voxels, overlap=0.474) Left_VentralDC (28): linear fit = 1.07 x + 0.0 (1713 voxels, peak = 93), gca=92.7 gca peak = 0.14975 (24) mri peak = 0.16216 (25) gca peak = 0.19357 (14) mri peak = 0.17613 (21) Fourth_Ventricle (15): linear fit = 1.40 x + 0.0 (148 voxels, overlap=0.280) Fourth_Ventricle (15): linear fit = 1.40 x + 0.0 (148 voxels, peak = 20), gca=19.7 gca peak Unknown = 0.94835 ( 0) gca peak Left_Inf_Lat_Vent = 0.16825 (27) gca peak Left_Thalamus = 1.00000 (94) gca peak Third_Ventricle = 0.14975 (24) gca peak CSF = 0.23379 (36) gca peak Left_Accumbens_area = 0.70037 (62) gca peak Left_undetermined = 1.00000 (26) gca peak Left_vessel = 0.75997 (52) gca peak Left_choroid_plexus = 0.12089 (35) gca peak Right_Inf_Lat_Vent = 0.24655 (23) gca peak Right_Accumbens_area = 0.45042 (65) gca peak Right_vessel = 0.82168 (52) gca peak Right_choroid_plexus = 0.14516 (37) gca peak Fifth_Ventricle = 0.65475 (32) gca peak WM_hypointensities = 0.07854 (76) gca peak non_WM_hypointensities = 0.08491 (43) gca peak Optic_Chiasm = 0.71127 (75) mri_ca_label: could not open intensity tracking file aseg.auto_noCCseg.label_intensities.txt not using caudate to estimate GM means estimating mean gm scale to be 1.12 x + 0.0 estimating mean wm scale to be 1.03 x + 0.0 estimating mean csf scale to be 1.30 x + 0.0 Left_Pallidum too bright - rescaling by 0.988 (from 1.095) to 103.9 (was 105.1) saving intensity scales to aseg.auto_noCCseg.label_intensities.txt Permission denied Linux c187-024.frontera.tacc.utexas.edu 3.10.0-1160.45.1.el7.x86_64 #1 SMP Wed Oct 13 17:20:51 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-mdd11 exited with ERRORS at Tue Mar 1 13:59:02 UTC 2022

For more details, see the log file /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds000171-fmriprep/sourcedata/freesurfer/sub-mdd11/scripts/recon-all.log To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Standard error:

Return code: 1

[jbwexler]

I got this issue again when trying to rerun a failed subject from ds000117 (different dataset from above). It seems we need to unlock more freesurfer files than just "$fs_path"/sub*/scripts/. The question is whether we can just unlock a few more files or if we need to unlock the entire fs directory. Though the average size of a freesurfer directory seems to only be about 250mb per subject, so the worst case scenario isn't so bad.

[jbwexler]

We resolved this by unlocking the entire freesurfer sub directory for any subject that's being run.