ds001497-fmriprep: "OSError: [Errno 39] Directory not empty: 'label'"

[jbwexler]

This happened for all the subjects I tested, which is not surprising since it seems to result from an issue with fsaverage. This is probably not relevant, but I should note that this run was with --resource-monitor, whereas the last run (which was successful) was without --resource-monitor. Compared to the last run, this datasets's fsaverage is missing the following files:

Only in fsaverage/label: lh.BA1_exvivo.label Only in fsaverage/label: lh.BA1_exvivo.thresh.label Only in fsaverage/label: lh.BA2_exvivo.label Only in fsaverage/label: lh.BA2_exvivo.thresh.label Only in fsaverage/label: lh.BA3a_exvivo.label Only in fsaverage/label: lh.BA3a_exvivo.thresh.label Only in fsaverage/label: lh.BA3b_exvivo.label Only in fsaverage/label: lh.BA3b_exvivo.thresh.label Only in fsaverage/label: lh.BA44_exvivo.label Only in fsaverage/label: lh.BA44_exvivo.thresh.label Only in fsaverage/label: lh.BA45_exvivo.label Only in fsaverage/label: lh.BA45_exvivo.thresh.label Only in fsaverage/label: lh.BA4a_exvivo.label Only in fsaverage/label: lh.BA4a_exvivo.thresh.label Only in fsaverage/label: lh.BA4p_exvivo.label Only in fsaverage/label: lh.BA4p_exvivo.thresh.label Only in fsaverage/label: lh.BA6_exvivo.label Only in fsaverage/label: lh.BA6_exvivo.thresh.label Only in fsaverage/label: lh.FG1.mpm.vpnl.label Only in fsaverage/label: lh.FG2.mpm.vpnl.label Only in fsaverage/label: lh.FG3.mpm.vpnl.label Only in fsaverage/label: lh.FG4.mpm.vpnl.label Only in fsaverage/label: lh.MT_exvivo.label Only in fsaverage/label: lh.MT_exvivo.thresh.label Only in fsaverage/label: Yeo_Brainmap_fsaverage_README

Here's the crashfile:

Node: fmriprep_23_0_wf.fsdir_run_20230508_204300_64bba3e1_c651_43b2_9368_fc75c16c49c1 Working directory: /scratch1/03201/jbwexler/work_dir/fmriprep/ds001497_sub-01/fmriprep_23_0_wf/fsdir_run_20230508_204300_64bba3e1_c651_43b2_9368_fc75c16c49c1

Node inputs:

derivatives = /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds001497-fmriprep freesurfer_home = /opt/freesurfer minimum_fs_version = 7.0.0 overwrite_fsaverage = False spaces = ['fsaverage5', 'fsaverage'] subjects_dir = /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/fmriprep/ds001497-fmriprep/sourcedata/freesurfer

Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers self.procs[jobid].run(updatehash=updatehash) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface return self._run_command(execute) File "/opt/conda/lib/python3.9/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsdir_run_20230508_204300_64bba3e1_c651_43b2_9368_fc75c16c49c1.

Traceback: Traceback (most recent call last): File "/opt/conda/lib/python3.9/site-packages/nipype/interfaces/base/core.py", line 398, in run runtime = self._run_interface(runtime) File "/opt/conda/lib/python3.9/site-packages/niworkflows/interfaces/bids.py", line 957, in _run_interface shutil.rmtree(dest) File "/opt/conda/lib/python3.9/shutil.py", line 732, in rmtree _rmtree_safe_fd(fd, path, onerror) File "/opt/conda/lib/python3.9/shutil.py", line 671, in _rmtree_safe_fd onerror(os.rmdir, fullname, sys.exc_info()) File "/opt/conda/lib/python3.9/shutil.py", line 669, in _rmtree_safe_fd os.rmdir(entry.name, dir_fd=topfd) OSError: [Errno 39] Directory not empty: 'label'

[jbwexler]

Yikes, I just noticed this occurred in several other datasets that I ran. I hadn't noticed because fmriprep still tells me it ran successfully.

I just submitted a ticket to TACC in case they are deleting the fsaverage files.

[jbwexler]

The fact that this person also had an issue with fsaverage/label makes me think it's an fmriprep issue not a TACC issue.

[jbwexler]

Tacc just confirmed that they haven't purged any files since last spring.

[effigies]

You're not running multiple processes in the same dataset? It's hard to see how this could happen just from the code, if we can't get a reliable reproduction.

[jbwexler]

Multiple processes of fmriprep? I'm running subjects in parallel using reproman, nothing has changed with that as far as I'm aware.

[jbwexler]

Resolved by copying in fsaverage prior to running fmriprep.