search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
638 stars 294 forks source link

fmriPrep 21.0.1 .h5 files no longer read by antsApplyTransforms? #2756

Open taforest opened 2 years ago

taforest commented 2 years ago

Hi,

I'm trying to move the aparcaseg file created by fmriprep into MNI152NLin2009cAsym to use the freesurfer ROIs in standard space. Using an older version of fmriprep (20.2.0) I had no trouble using the .h5 files saved automatically by fmriprep in this antsApplyTransform call (removed the absolute path names, but file names all the same):

antsApplyTransforms \ -d 3 \ -i /derivatives/sub-101/anat/sub-101_desc-aparcaseg_dseg.nii.gz \ -r /derivatives/sub-101/anat/sub-101_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz \ -t /derivatives/sub-101/anat/sub-101_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5 \ -n nearestNeighbor \ -o /masks/sub-101/sub-101_desc-aparcaseg_dseg_MNI.nii.gz

But, after running fmriprep 21.0.1 instead, this same call is resulting in the following error (the output file structure is slightly different in this fmriprep version):

HDF5-DIAG: Error detected in HDF5 (1.8.17) thread 0:

000: /software/ANTs/ITKv4/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5D.c line 358 in H5Dopen2(): not found

major: Dataset
minor: Object not found

001: /software/ANTs/ITKv4/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Gloc.c line 430 in H5G_loc_find(): can't find object

major: Symbol table
minor: Object not found

002: /software/ANTs/ITKv4/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed

major: Symbol table
minor: Object not found

003: /software/ANTs/ITKv4/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed

major: Symbol table
minor: Callback failed

004: /software/ANTs/ITKv4/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Gloc.c line 385 in H5G_loc_find_cb(): object 'TranformFixedParameters' doesn't exist

major: Symbol table
minor: Object not found

Transform reader for /derivatives/sub-101/anat/sub-101_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5 caught an ITK exception:

itk::ExceptionObject (0x3d27a10) Location: "unknown" File: /software/ANTs/ITKv4/Modules/IO/TransformHDF5/include/itkHDF5TransformIO.hxx Line: 356 Description: itk::ERROR: HDF5TransformIOTemplate(0x3b940d0): H5Dopen2 failed

Can't read initial transform /landlab/statlearn/2021/realdata/derivatives/sub-101/anat/sub-101_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5

Some threads make it sound like theres a version compatibility issue with ants and frmiprep now-- if so, which version of ants is the one that should be used with this fmriprep version (can't find this info on the documentation)? Has the output structure of the .h5 files has changed between these fmriprep versions? Sorry if this has been answered elsewhere! I'd appreciate any guidance and I'm happy to provide additional information if needed. Thank you!!

effigies commented 2 years ago

The same version of ANTs (2.3.4) is in both Docker images. I'm not sure when we last upgraded ANTs.

Is it possible that the file is just malformed and rerunning would help? Or are you seeing this consistently?

Some threads make it sound like theres a version compatibility issue with ants and frmiprep now-

Can you link to these threads? This is the first I'm hearing of it.

taforest commented 2 years ago

Seems to be all my subjects.

Similar thread here (although for this user the fmriprep output worked): https://github.com/PennLINC/qsiprep/issues/128

And something potentially similar: https://github.com/SimpleITK/SimpleITK/issues/844

effigies commented 2 years ago

Interesting. I guess there are incompatibilities in ANTs versions. Which is a bit strange, since they haven't had a release in over a year, so we're only one official version behind. I would suggest just using ANTs v2.3.4 or earlier to work with fMRIPrep outputs, for now.

toomanycats commented 2 years ago

I hit the same problem today in fmriPrep 21.0.2 using Docker --> singularity build. 

RuntimeError: Command:
antsApplyTransforms --default-value 0 --float 1 --input /work/fmriprep_wf/single_subject_DPOP022_wf/func_preproc_ses_1_task_rest_run_7_wf/bold_split/vol0001.nii.gz --interpolation LanczosWindowedSinc --output /work/fmriprep_wf/single_subject_DPOP022_wf/func_preproc_ses_1_task_rest_run_7_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/vol0001_xform-00001.nii.gz --reference-image /work/fmriprep_wf/single_subject_DPOP022_wf/func_preproc_ses_1_task_rest_run_7_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/gen_ref/tpl-MNI152NLin2009cAsym_res-02_T1w_reference.nii.gz --transform /work/fmriprep_wf/single_subject_DPOP022_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration/ants_t1_to_mniComposite.h5 --transform /work/fmriprep_wf/single_subject_DPOP022_wf/func_preproc_ses_1_task_rest_run_7_wf/bold_reg_wf/bbreg_wf/concat_xfm/out_fwd.tfm --transform identity --transform /work/fmriprep_wf/single_subject_DPOP022_wf/func_preproc_ses_1_task_rest_run_7_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/bold_to_std_transform/tmp-r5u5c1tz/mat2itk_pos-003_xfm-00001.txt
Standard output:

Standard error:
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 0:
  #000: /src/ants/build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5F.c line 509 in itk_H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: /src/ants/build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Fint.c line 1400 in itk_H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: /src/ants/build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Fint.c line 1615 in itk_H5F_open(): unable to lock the file
    major: File accessibilty
    minor: Unable to open file
  #003: /src/ants/build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FD.c line 1640 in itk_H5FD_lock(): driver lock request failed
    major: Virtual File Layer
    minor: Can't update object
  #004: /src/ants/build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDsec2.c line 941 in H5FD_sec2_lock(): unable to lock file, errno = 5, error message = 'Input/output error'
    major: File accessibilty
    minor: Bad file ID accessed
Transform reader for /work/fmriprep_wf/single_subject_DPOP022_wf/anat_preproc_wf/anat_norm_wf/_template_MNI152NLin2009cAsym/registration/ants_t1_to_mniComposite.h5 caught an ITK exception:

itk::ExceptionObject (0x39845d0)
Location: "unknown" 
File: /src/ants/build/ITKv5/Modules/IO/TransformHDF5/src/itkHDF5TransformIO.cxx
Line: 363
Description: itk::ERROR: HDF5TransformIOTemplate(0x3979880): H5Fopen failed

Return code: 1
toomanycats commented 2 years ago

When I run my container again with the work directory bound to /tmp this issue went away. Part of the error message says a file lock is not obtained. I wonder if NFS is the culprit ??

It's difficult to run this all in /tmp for all my subjects. Any help is appreciated.

I tried running the sing container from inside and waiting for the error again. It did happen, but when I run the applyTransform command from previous stack trace, one of the files was not found. Running the command again without the offending path and but including the hf5 file was successful.

toomanycats commented 1 year ago

I've been doing more research on this issue. I believe it has to do with how the Linux kernel uses file locks on an NFS mounted file system. This will probably only be solved with an update to how ITK issues a lock.