Open WangYunHong98 opened 1 month ago
240724-20:22:13,885 nipype.workflow INFO:
[Node] Executing "fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad" <niworkflows.interfaces.bids.BIDSFreeSurferDir>
240724-20:22:13,939 nipype.workflow INFO:
[Node] Finished "fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad", elapsed time 0.046097s.
240724-20:22:13,939 nipype.workflow DEBUG:
Saving results file: '/work/fmriprep_24_0_wf/fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad/result_fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad.pklz'
240724-20:22:13,939 nipype.workflow WARNING:
Storing result file without outputs
240724-20:22:13,947 nipype.workflow WARNING:
[Node] Error on "fmriprep_24_0_wf.fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad" (/work/fmriprep_24_0_wf/fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad)
240724-20:22:13,953 nipype.workflow DEBUG:
Clearing 0 from queue
240724-20:22:13,953 nipype.utils DEBUG:
Loading pkl: /work/fmriprep_24_0_wf/fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad/result_fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad.pklz
240724-20:22:13,966 nipype.workflow ERROR:
Node fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad failed to run on host idw01.
240724-20:22:13,967 nipype.workflow ERROR:
Saving crash info to /output/logs/crash-20240724-202213-wyh-fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad-0750bb4e-85fc-437e-a7a8-60740dbafebb.txt
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
self.procs[jobid].run(updatehash=updatehash)
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
result = self._run_interface(execute=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
return self._run_command(execute)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad.
Traceback:
Traceback (most recent call last):
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/niworkflows/interfaces/bids.py", line 960, in _run_interface
shutil.rmtree(dest)
File "/opt/conda/envs/fmriprep/lib/python3.11/shutil.py", line 752, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/opt/conda/envs/fmriprep/lib/python3.11/shutil.py", line 683, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/opt/conda/envs/fmriprep/lib/python3.11/shutil.py", line 681, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'label'
240724-20:22:13,981 nipype.workflow INFO:
[Job 0] Completed (fmriprep_24_0_wf.fsdir_run_20240724_202139_7f3a5ef2_8f8b_4828_b5fa_f486b97008ad).
There's either a permissions issue or a concurrency issue. Are you running multiple subjects simultaneously with the same FreeSurfer directory?
Hi @effigies
I possibly run multiple subjects because I submitted the slurm
job using this code sbatch --array=1-$(( $( wc -l ${PROJECT}/BIDS_heudiconv/participants.tsv | cut -f1 -d' ' ) - 1 ))%20 fmriprep.slurm
So what's happening is that you have an fsaverage
directory from an older version of FreeSurfer. To keep FreeSurfer working correctly, we delete and replace with the bundled copy. If you have multiple copies of fMRIPrep doing this at the same time, you will get race conditions.
There's no real plausible way to protect against this in code. It's up to users not to call fMRIPrep in parallel in this situation.
Hi @effigies
It occurs an error later.
ERROR: Label BA1_exvivo does not exist in SUBJECTS_DIR fsaverage! The fsaverage link probably points to an older freesurfer version
This error is very strange. Sometimes I'm running fmriprep and I don't get this error, sometimes I run it and it keeps coming up. Is there any way to fix this? For example, submit a job first. After this job is completed, I submit the parallel array job.
That sounds like a recurrence of the other issue you opened: #3258.
@tsalo
Yes, it's an old problem. I tested it, and if I submit a job array for one subject, fmriprep (version-24.0.1) runs successfully with doing surface reconstruction. But if I submit a job array for multiple subjects, I get the error I mentioned above.
By the way, the --containall
was added to the script.
So, I'm ready to try this way:
--fs-subjects-dir /output/freesurfer/$subject
-w /work/$subject
Prepare seperate directories for each subject.
But I'm not sure it's the right thing to do. What do you think? @effigies
What happened?
I used fmriprep-24.0.0 on
slurm
to process my MRI data, but the outputs were incomplete.For example,
freesurfer
directory was empty and the output of--output-space
I specified was not produced in thefunc
andanat
directory.I uploaded the log file. Please check it out fmriprep-30459_6.json
What command did you use?
What version of fMRIPrep are you running?
24.0.0
How are you running fMRIPrep?
Singularity
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
Additional information / screenshots
There are outputs in
func
directory