Crash when running xcp_d 0.1.3 (nodes left_sphere_raw_mris, right_sphere_raw_mris)

danella commented 2 years ago

Summary

When running xcp-d (using output from flywheel fmriprep version 20.2.6), several crash reports are generated. All outputs seem to be present, but I am not sure if they are valid, given these crashes.

Additional details

xcp_d version: 0.1.3

What were you trying to do?

Running xcp-d 0.1.3 on the cluster using the following code:

singularity run --cleanenv -B /ix/dhafeman:/home/xcp_d /ix/dhafeman/singularity/xcp_d-0.1.3.simg /ix/dhafeman/BDLONG/Preproc/fmriprep /ix/dhafeman/BDLONG/Preproc participant -f 0.1 -w /ix/dhafeman/BDLONG/Work --smoothing 8 --despike -p 36P --lower-bpf .009 --upper-bpf .08 --motion-filter-type notch --band-stop-min 12 --band-stop-max 20 --participant_label 400

What did you expect to happen?

Complete analysis without crash reports

What actually happened?

Complete analysis, but with the following crash reports

crash-20221006-201530-dmh123-left_sphere_raw_mris-663754d8-f1e5-496e-8f7e-95216afa204e.txt crash-20221006-201532-dmh123-right_sphere_raw_mris-fefa4473-de8b-4569-a8c7-824dd1dc1daa.txt crash-20221007-113941-dmh123-rh_surface_apply_warpfield.aI.a0-8475e25d-9a75-4df5-a732-96bf31dfab39.txt crash-20221007-113941-dmh123-rh_surface_apply_warpfield.aI.a1-c170ef34-cab3-4e75-a5ff-0600303f905c.txt crash-20221007-113945-dmh123-lh_surface_apply_warpfield.aI.a1-65b25e7d-20c2-4918-8555-aa642cbb58e9.txt

Here is an example:

Node: xcpd_wf.single_subject_400_wf.anatomical_wf.left_sphere_raw_mris
Working directory: /home/xcp_d/BDLONG/Test_cifti/Work_test/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris

Node inputs:

annot_file = <undefined>
args = <undefined>
dataarray_num = <undefined>
environ = {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
functional_file = <undefined>
in_file = /home/xcp_d/BDLONG/Test_cifti/Preproc_test/fmriprep/sourcedata/freesurfer/sub-400/surf/lh.sphere.reg
label_file = <undefined>
labelstats_outfile = <undefined>
normal = <undefined>
origname = <undefined>
out_datatype = gii
out_file = <undefined>
parcstats_file = <undefined>
patch = <undefined>
rescale = <undefined>
scalarcurv_file = <undefined>
scale = <undefined>
subjects_dir = /opt/freesurfer/subjects
talairachxfm_subjid = <undefined>
to_scanner = <undefined>
to_tkr = <undefined>
vertex = <undefined>
xyz_ascii = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node left_sphere_raw_mris.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'converted' trait of a MRIsConvertOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/xcp_d/BDLONG/Test_cifti/Work_test/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/lh.sphere.reg_converted.gii' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/home/xcp_d/BDLONG/Test_cifti/Work_test/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/lh.sphere.reg_converted.gii' for output 'converted' of a MRIsConvert interface

Reproducing the bug

I posted on neurostars and received the following advice: https://neurostars.org/t/crash-when-running-xcp-d-0-1-3-nodes-left-sphere-raw-mris-and-right-sphere-raw-mris/23704

I have tried the following (same error occurred) -Running using a new version of fmriprep (21.0.2) -Running using the cifti option

Thanks in advance for your help! Best, Danella

-->

tsalo commented 2 years ago

Thanks for raising this.

It looks like xcp_d is looking for the required file at /ix/dhafeman/BDLONG/Test_cifti/Preproc_test/fmriprep/sourcedata/freesurfer/sub-400/surf/lh.sphere.reg. Does that file exist?

danella commented 2 years ago

Thanks for your quick response! Yes, it exists.

[dmh123@psych ~]$ ls /ix/dhafeman/BDLONG/Test_cifti/Preproc_test/fmriprep/sourcedata/freesurfer/sub-400/surf -1
lh.area
lh.area.mid
lh.area.pial
lh.avg_curv
lh.curv
lh.curv.pial
lh.defect_borders
lh.defect_chull
lh.defect_labels
lh.inflated
lh.inflated.H
lh.inflated.K
lh.inflated.nofix
lh.jacobian_white
lh.midthickness
lh.orig
lh.orig.nofix
lh.pial
lh.qsphere.nofix
lh.smoothwm
lh.smoothwm.BE.crv
lh.smoothwm.C.crv
lh.smoothwm.FI.crv
lh.smoothwm.H.crv
lh.smoothwm.K1.crv
lh.smoothwm.K2.crv
lh.smoothwm.K.crv
lh.smoothwm.nofix
lh.smoothwm.S.crv
lh.sphere
lh.sphere.reg
lh.sulc
lh.thickness
lh.volume
lh.w-g.pct.mgh
lh.white
lh.white.H
lh.white.K
lh.white.preaparc
lh.white.preaparc.H
lh.white.preaparc.K
lh.woT2.pial
rh.area
rh.area.mid
rh.area.pial
rh.avg_curv
rh.curv
rh.curv.pial
rh.defect_borders
rh.defect_chull
rh.defect_labels
rh.inflated
rh.inflated.H
rh.inflated.K
rh.inflated.nofix
rh.jacobian_white
rh.midthickness
rh.orig
rh.orig.nofix
rh.pial
rh.qsphere.nofix
rh.smoothwm
rh.smoothwm.BE.crv
rh.smoothwm.C.crv
rh.smoothwm.FI.crv
rh.smoothwm.H.crv
rh.smoothwm.K1.crv
rh.smoothwm.K2.crv
rh.smoothwm.K.crv
rh.smoothwm.nofix
rh.smoothwm.S.crv
rh.sphere
rh.sphere.reg
rh.sulc
rh.thickness
rh.volume
rh.w-g.pct.mgh
rh.white
rh.white.H
rh.white.K
rh.white.preaparc
rh.white.preaparc.H
rh.white.preaparc.K
rh.woT2.pial

tsalo commented 2 years ago

AFAICT, MRIsConvert is failing on that file, which would explain why the output file is missing. Can you share the contents of /ix/dhafeman/BDLONG/Test_cifti/Work_test/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/?

On an unrelated note, in your xcpd call, it looks like you bound /ix/dhafeman to /home/xcpd, but the paths you put in the call are all from /ix/dhafeman (e.g., /ix/dhafeman/BDLONG/Preproc/fmriprep). Shouldn't the image not be able to find anything in /ix/dhafeman? It doesn't seem like that's causing any problems, since then xcpd would probably just not find any files, but it does seem weird to me.

Could you try running the following instead? Basically, I switched out /ix/dhafeman with /home/xcp_d. Maybe the paths are causing a problem that I don't understand.

singularity run --cleanenv \
    -B /ix/dhafeman:/home/xcp_d \
    /ix/dhafeman/singularity/xcp_d-0.1.3.simg \
    /home/xcp_d/BDLONG/Preproc/fmriprep \
    /home/xcp_d/BDLONG/Preproc \
    participant -f 0.1 \
    -w /home/xcp_d/BDLONG/Work \
    --smoothing 8 --despike -p 36P --lower-bpf .009 --upper-bpf .08 \
    --motion-filter-type notch --band-stop-min 12 --band-stop-max 20 --participant_label 400

danella commented 2 years ago

Oops, I got the same feedback on neurostars, and did change this in the most recent iteration of this code:

singularity run --cleanenv -B /ix/dhafeman:/home/xcp_d /ix/dhafeman/singularity/xcp_d-0.1.3.simg /home/xcp_d/BDLONG/Test_cifti/Preproc_test/fmriprep /home/xcp_d/BDLONG/Test_cifti/Preproc_test participant -f 0.1 -w /home/xcp_d/BDLONG/Test_cifti/Work_test --smoothing 8 --despike -p 36P --lower-bpf .009 --upper-bpf .08 --motion-filter-type notch --band-stop-min 12 --band-stop-max 20 --participant_label 400

Unfortunately, this didn't solve the problem.

And here are the contents of that directory:

[dmh123@psych ~]$ ls /ix/dhafeman/BDLONG/Test_cifti/Work_test/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/ -1
command.txt
_inputs.pklz
_node.pklz
_report
result_left_sphere_raw_mris.pklz

tsalo commented 2 years ago

Ah, okay. Can you post the contents of command.txt and _report/report.rst?

danella commented 2 years ago

command.txt:

mris_convert /home/xcp_d/BDLONG/Test_cifti/Preproc_test/fmriprep/sourcedata/freesurfer/sub-400/surf/lh.sphere.reg /home/xcp_d/BDLONG/Test_cifti/Work_test/xcpd_wf/single_subject_400_wf/anatomical_wf/left_sphere_raw_mris/lh.sphere.reg_converted.gii

_report/report.rst

Node: single_subject_400_wf (anatomical_wf (left_sphere_raw_mris (freesurfer)
=============================================================================

 Hierarchy : xcpd_wf.single_subject_400_wf.anatomical_wf.left_sphere_raw_mris
 Exec ID : left_sphere_raw_mris

Original Inputs
---------------

* annot_file : <undefined>
* args : <undefined>
* dataarray_num : <undefined>
* environ : {'SUBJECTS_DIR': '/opt/freesurfer/subjects'}
* functional_file : <undefined>
* in_file : /home/xcp_d/BDLONG/Test_cifti/Preproc_test/fmriprep/sourcedata/freesurfer/sub-400/surf/lh.sphere.reg
* label_file : <undefined>
* labelstats_outfile : <undefined>
* normal : <undefined>
* origname : <undefined>
* out_datatype : gii
* out_file : <undefined>
* parcstats_file : <undefined>
* patch : <undefined>
* rescale : <undefined>
* scalarcurv_file : <undefined>
* scale : <undefined>
* subjects_dir : /opt/freesurfer/subjects
* talairachxfm_subjid : <undefined>
* to_scanner : <undefined>
* to_tkr : <undefined>
* vertex : <undefined>
* xyz_ascii : <undefined>

madisoth commented 2 years ago

Hi all,

At a glance the issue looks like FreeSurfer mris_convert is failing because the expected FreeSurfer license file is not bound to /opt/freesurfer/license.txt. We do need to document that requirement more clearly; it became a requirement around 0.1.0 or so.

Binding that file should avert that crash, but there are still a couple issues to be aware of:

In xcp_d 0.1.3, the resampling of surfaces to the fsLR32k standard is incorrect (BOLD dtseries will appear misaligned on the surfaces and/or surfaces will be misaligned with the MNI152NLin6Asym volumes). This is addressed by https://github.com/PennLINC/xcp_d/pull/496 and should be fixed in a upcoming release.
In fMRIPrep versions < 22.0.0, the surf.gii derivatives are misaligned with the T1w volumes due to the --to-scanner flag being omitted when converting from FS to GIfTI format with mris_convert. I haven't tested myself, but instead of reprocessing your data with a newer fMRIPrep version, it may be sufficient to just re-convert the surfaces in the FreeSurfer directory with mris_convert --to-scanner and use that to replace your existing fMRIPrep .surf.gii derivatives.

danella commented 2 years ago

Oh, I see! That's great! I am trying again with an extra -B /ix/dhafeman/license.txt:/opt/freesurfer/license.tx

As far as the issues that you raised, I am not planning on using any of the cifti outputs from fmriprep. (I had just added that option in, since this was suggested on neurostars as a possible fix to this crash.) If I will only be working in volume space, are these additional issues problematic?

Thank yoU!

danella commented 2 years ago

This resolved the first two errors: crash-20221006-201530-dmh123-left_sphere_raw_mris-663754d8-f1e5-496e-8f7e-95216afa204e.txt crash-20221006-201532-dmh123-right_sphere_raw_mris-fefa4473-de8b-4569-a8c7-824dd1dc1daa.txt

But I am still seeing these errors: crash-20221007-113941-dmh123-rh_surface_apply_warpfield.aI.a0-8475e25d-9a75-4df5-a732-96bf31dfab39.txt crash-20221007-113941-dmh123-rh_surface_apply_warpfield.aI.a1-c170ef34-cab3-4e75-a5ff-0600303f905c.txt crash-20221007-113945-dmh123-lh_surface_apply_warpfield.aI.a1-65b25e7d-20c2-4918-8555-aa642cbb58e9.txt

Here is the rh_surface crash file

ode: xcpd_wf.single_subject_400_wf.anatomical_wf.rh_surface_apply_warpfield
Working directory: /home/xcp_d/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/_in_file_..home..xcp_d..BDLONG..Preproc..fmriprep..sub-400..ses-01..anat..sub-400_ses-01_acq-mprage_hemi-R_pial.surf.gii/rh_surface_apply_warpfield

Node inputs:

args = <undefined>
environ = {}
forward_warp = <undefined>
in_file = /home/xcp_d/BDLONG/Preproc/fmriprep/sub-400/ses-01/anat/sub-400_ses-01_acq-mprage_hemi-R_pial.surf.gii
out_file = <undefined>
warpfield = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node rh_surface_apply_warpfield.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_file' trait of an ApplyWarpfieldOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/xcp_d/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/_in_file_..home..xcp_d..BDLONG..Preproc..fmriprep..sub-400..ses-01..anat..sub-400_ses-01_acq-mprage_hemi-R_pial.surf.gii/rh_surface_apply_warpfield/sub-400_ses-01_acq-mprage_hemi-R_pial.surf-MNIwarped.surf.gii' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/home/xcp_d/BDLONG/Work/xcpd_wf/single_subject_400_wf/anatomical_wf/_in_file_..home..xcp_d..BDLONG..Preproc..fmriprep..sub-400..ses-01..anat..sub-400_ses-01_acq-mprage_hemi-R_pial.surf.gii/rh_surface_apply_warpfield/sub-400_ses-01_acq-mprage_hemi-R_pial.surf-MNIwarped.surf.gii' for output 'out_file' of a ApplyWarpfield interface

I double-checked, and sub-400_ses-01_acq-mprage_hemi-R_pial.surf.gii is in the right place. Any further suggestions? Also, if my results are valid even with this error, I am ok with that too!

madisoth commented 2 years ago

The warpfield transform may still be failing because the native surfaces extend outside the FOV of the warpfield; that is another issue addressed by https://github.com/PennLINC/xcp_d/pull/496. But those errors won't affect the processing of the BOLD timeseries; the functional processing workflow is independent on the outputs of the anatomical workflow.

kahinimehta commented 2 years ago

Closing this issue for now, feel free to re-open if there are any other problems!

PennLINC / xcp_d