search

PennLINC / xcp_d

Post-processing of fMRIPrep, NiBabies, and HCP outputs
https://xcp-d.readthedocs.io
BSD 3-Clause "New" or "Revised" License
78 stars 26 forks source link

Module qc_report_wf has no input #1310

Closed Xinquan-Lu closed 1 week ago

Xinquan-Lu commented 1 week ago

Summary

Module qc_report_wf has no input. I checked the fmriprep and confirmed I have minimal input to run the Cifti post process. But it always reported that qc_report_wf did not have some input found. My data was collected as Nifit type. I use fmriprep to generate Cifit files.

Additional details

What were you trying to do?

Try to run the Cifit type post process.

What did you expect to happen?

xcp-d finished successfully.

What actually happened?

Module qc_report_wf has no input.

error output is like this:

Exception: Some connections were not found
Module qc_report_wf has no input called inputnode.name_source

Module qc_report_wf has no input called inputnode.preprocessed_bold

Module qc_report_wf has no input called inputnode.dummy_scans

Module qc_report_wf has no input called inputnode.fmriprep_confounds_file

Module qc_report_wf has no input called inputnode.temporal_mask

Module qc_report_wf has no input called inputnode.filtered_motion

Module qc_report_wf has no input called inputnode.denoised_interpolated_bold

Module qc_report_wf has no input called inputnode.censored_denoised_bold

Reproducing the bug

fmriprep:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=20G
#SBATCH -p q_fat_c
#SBATCH --qos=high_c

module load singularity
module load freesurfer/7.1.1

#User inputs:
subj=$1
path=/ibmgpfs/cuizaixu_lab/luxinquan/CCR
bids_dir=$path/BIDS
work_dir=$path/z_wd_fMRIprep
output_dir=$path/fMRIprep
if [ ! -d "$work_dir" ]; then
mkdir $work_dir
fi
if [ -d "$work_dir/$subj" ]; then
rm -rf $work_dir/$subj
fi
if [ ! -d "$work_dir/$subj" ]; then
mkdir $work_dir/$subj
fi
if [ ! -d "$output_dir" ]; then
mkdir $output_dir
fi
fs_license=/ibmgpfs/cuizaixu_lab/luxinquan/tool/freesurfer
templateflow=/ibmgpfs/cuizaixu_lab/luxinquan/tool/templateflow
#Run fmriprep
echo ""
echo "Running fmriprep on participant: $subj"
echo ""
export SINGULARITYENV_TEMPLATEFLOW_HOME=$templateflow
unset PYTHONPATH;
singularity run --cleanenv \
    -B $work_dir/$subj:/work_dir \
    -B $bids_dir:/bids_dir \
    -B $output_dir:/output_dir \
    -B $fs_license:/fs_license \
    -B $templateflow:$templateflow \
    /ibmgpfs/cuizaixu_lab/luxinquan/apps/singularity/fmriprep.simg \
    /bids_dir /output_dir participant -w /work_dir \
    --n-cpus 2 --omp-nthreads 2 --mem-mb 40G \
    --participant_label ${subj} \
    --fs-license-file /fs_license/license.txt \
    --skip_bids_validation \
    --output-spaces T1w MNI152NLin6Asym:res-2 MNI152NLin2009cAsym:res-2 \
    --return-all-components \
    --cifti-output 91k \
    --stop-on-first-crash \
    --resource-monitor \
    --notrack
rm -rf $work_dir/$subj

xcp-d:

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=20G
#SBATCH -p q_fat_c
#SBATCH --qos=high_c

module load singularity

subjID=$1
fmriprep_dir=/ibmgpfs/cuizaixu_lab/luxinquan/CCR/BIDS/derivatives/fMRIprep
output_dir=/ibmgpfs/cuizaixu_lab/luxinquan/CCR/xcpd
work_dir=/ibmgpfs/cuizaixu_lab/luxinquan/CCR/z_wd_xcpd
singularity_image=/ibmgpfs/cuizaixu_lab/luxinquan/apps/singularity/xcp_d-0.9.1.simg
fs_license=/ibmgpfs/cuizaixu_lab/luxinquan/tool/freesurfer
templateflow=/ibmgpfs/cuizaixu_lab/luxinquan/tool/templateflow

mkdir -p "$output_dir"
mkdir -p "$work_dir/$subjID"

if [ -d "$work_dir/$subjID" ]; then
    rm -rf "$work_dir/$subjID"
fi
mkdir -p "$work_dir/$subjID"

export SINGULARITYENV_TEMPLATEFLOW_HOME=$templateflow
unset PYTHONPATH;

echo ""
echo "Running XCP-D on participant: $subjID"
echo ""

singularity run --cleanenv \
    -B "$work_dir/$subjID":/work_dir \
    -B "$fmriprep_dir":/input \
    -B "$output_dir":/output \
    -B "$fs_license":/fs_license \
    -B $templateflow:$templateflow \
    ${singularity_image} \
    /input /output participant \
    --participant_label $subjID \
    --fs-license-file /fs_license/license.txt \
    --work-dir /work_dir \
    --mode none \
    --input-type fmriprep \
    --file-format cifti \
    --fd-thresh 0.2 \
    --despike n \
    --abcc-qc n \
    --combine-runs n \
    --motion-filter-type none \
    --output-type interpolated \
    --nuisance-regressors 36P \
    --linc-qc n \
    --warp-surfaces-native2std n \
    --resource-monitor \
    --notrack

rm -rf "$work_dir/$subjID"

xcpd_submitjob:

#!/bin/bash

bids_dir=/ibmgpfs/cuizaixu_lab/luxinquan/CCR/BIDS

rm -f subjList.txt

ls $bids_dir | grep sub- | sed 's/sub-//' > subjList.txt

mkdir -p z_report_xcpd

for subj in `cat subjList.txt`
do
    echo ""
    echo "Running XCP-D on participant: $subj"
    echo ""

    sbatch -J xcpd_${subj} -o z_report_xcpd/${subj}_out.txt \
        -e z_report_xcpd/${subj}_error.txt \
        xcpd.sh $subj
done

-->

tsalo commented 1 week ago

@Xinquan-Lu can you try updating to 0.10.0rc3 and re-running?

Xinquan-Lu commented 1 week ago

I've tried, and the bug is almost the same. Here is the output.

2024-11-04 07:40:53,079 [ WARNING] WARNING: The sentry_sdk.metrics module is deprecated and will be removed in the next major release. Sentry will reject all metrics sent after October 7, 2024. Learn more: https://sentry.zendesk.com/hc/en-us/articles/26369339769883-Upcoming-API-Changes-to-Metrics
2024-11-04 07:40:53,088 [ WARNING] WARNING: `sentry_sdk.Hub` is deprecated and will be removed in a future major release. Please consult our 1.x to 2.x migration guide for details on how to migrate `Hub` usage to the new API: https://docs.sentry.io/platforms/python/migration/1.x-to-2.x
2024-11-04 07:41:17,235 [ WARNING] WARNING: The sentry_sdk.metrics module is deprecated and will be removed in the next major release. Sentry will reject all metrics sent after October 7, 2024. Learn more: https://sentry.zendesk.com/hc/en-us/articles/26369339769883-Upcoming-API-Changes-to-Metrics
2024-11-04 07:41:17,240 [ WARNING] WARNING: `sentry_sdk.Hub` is deprecated and will be removed in a future major release. Please consult our 1.x to 2.x migration guide for details on how to migrate `Hub` usage to the new API: https://docs.sentry.io/platforms/python/migration/1.x-to-2.x
2024-11-04 07:41:20,248 [IMPORTANT] Running XCP-D version 0.10.0rc3
2024-11-04 07:41:21,102 [IMPORTANT] Building XCP-D's workflow:
           * Preprocessing derivatives path: /input.
           * Participant list: ['20240614K96CCR001A'].
           * Run identifier: 20241104-074046_af53fa69-eeaf-440a-83f6-e4e86dc1434e.
           * Searching for derivatives and atlases: {'xcpdatlases': PosixPath('/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/data/atlases'), 'xcpd4s': PosixPath('/AtlasPack')}.
2024-11-04 07:41:22,005 [ WARNING] Both T1w and T2w found. Checking for T1w-space T2w.
2024-11-04 07:41:22,028 [ WARNING] No T1w-space T2w found. Checking for T2w-space T1w.
2024-11-04 07:41:22,049 [ WARNING] No T2w-space T1w found. Attempting T2w-only processing.
2024-11-04 07:41:22,070 [ WARNING] T2w-to-template transform not found. Attempting T1w-only processing.
2024-11-04 07:41:22,198 [IMPORTANT] Collected data:
anat_brainmask: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_space-MNI152NLin6Asym_res-2_desc-brain_mask.nii.gz
anat_to_template_xfm: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5
bold:
- /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-1_space-fsLR_den-91k_bold.dtseries.nii
- /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-2_space-fsLR_den-91k_bold.dtseries.nii
- /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-3_space-fsLR_den-91k_bold.dtseries.nii
t1w: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_desc-preproc_T1w.nii.gz
t2w: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_desc-preproc_T2w.nii.gz
template_to_anat_xfm: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_from-MNI152NLin6Asym_to-T1w_mode-image_xfm.h5

2024-11-04 07:41:22,485 [    INFO] No standard-space surfaces found.
2024-11-04 07:41:22,766 [IMPORTANT] Collected mesh files:
lh_pial_surf: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_hemi-L_pial.surf.gii
lh_subject_sphere: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_hemi-L_space-fsaverage_desc-reg_sphere.surf.gii
lh_wm_surf: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_hemi-L_white.surf.gii
rh_pial_surf: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_hemi-R_pial.surf.gii
rh_subject_sphere: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_hemi-R_space-fsaverage_desc-reg_sphere.surf.gii
rh_wm_surf: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_hemi-R_white.surf.gii

2024-11-04 07:41:22,766 [IMPORTANT] Assuming segmentation was performed with FreeSurfer.
2024-11-04 07:41:22,911 [IMPORTANT] Collected morphometry files:
cortical_thickness: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_space-fsLR_den-91k_thickness.dscalar.nii
cortical_thickness_corr: null
myelin: null
myelin_smoothed: null
sulcal_curv: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_space-fsLR_den-91k_curv.dscalar.nii
sulcal_depth: /input/sub-20240614K96CCR001A/anat/sub-20240614K96CCR001A_space-fsLR_den-91k_sulc.dscalar.nii

2024-11-04 07:41:23,757 [    INFO] Loading atlas: Glasser
2024-11-04 07:41:23,757 [    INFO] Loading atlas: Gordon
2024-11-04 07:41:23,757 [    INFO] Loading atlas: HCP
2024-11-04 07:41:23,757 [    INFO] Loading atlas: MIDB
2024-11-04 07:41:23,757 [    INFO] Loading atlas: MyersLabonte
2024-11-04 07:41:23,757 [    INFO] Loading atlas: Tian
2024-11-04 07:41:23,757 [    INFO] Loading atlas: 4S1056Parcels
2024-11-04 07:41:23,757 [    INFO] Loading atlas: 4S156Parcels
2024-11-04 07:41:23,757 [    INFO] Loading atlas: 4S256Parcels
2024-11-04 07:41:23,757 [    INFO] Loading atlas: 4S356Parcels
2024-11-04 07:41:23,757 [    INFO] Loading atlas: 4S456Parcels
2024-11-04 07:41:23,758 [    INFO] Loading atlas: 4S556Parcels
2024-11-04 07:41:23,758 [    INFO] Loading atlas: 4S656Parcels
2024-11-04 07:41:23,758 [    INFO] Loading atlas: 4S756Parcels
2024-11-04 07:41:23,758 [    INFO] Loading atlas: 4S856Parcels
2024-11-04 07:41:23,758 [    INFO] Loading atlas: 4S956Parcels
2024-11-04 07:41:32,159 [IMPORTANT] Collected run data for sub-20240614K96CCR001A_task-rest_run-1_space-fsLR_den-91k_bold.dtseries.nii:
boldref: /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-1_space-MNI152NLin6Asym_res-2_boldref.nii.gz
motion_file: /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-1_desc-confounds_timeseries.tsv
motion_json: /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-1_desc-confounds_timeseries.json
nifti_file: /input/sub-20240614K96CCR001A/func/sub-20240614K96CCR001A_task-rest_run-1_space-MNI152NLin6Asym_res-2_desc-preproc_bold.nii.gz

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/cli/workflow.py", line 100, in build_workflow
    retval["workflow"] = init_xcpd_wf()
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 81, in init_xcpd_wf
    single_subject_wf = init_single_subject_wf(subject_id)
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/base.py", line 469, in init_single_subject_wf
    postprocess_bold_wf = init_postprocess_bold_wf(
  File "/usr/local/miniconda/lib/python3.10/site-packages/xcp_d/workflows/bold/cifti.py", line 301, in init_postprocess_cifti_wf
    workflow.connect([
  File "/usr/local/miniconda/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 239, in connect
    raise Exception("\n".join(["Some connections were not found"] + infostr))
Exception: Some connections were not found
Module qc_report_wf has no input called inputnode.name_source

Module qc_report_wf has no input called inputnode.preprocessed_bold

Module qc_report_wf has no input called inputnode.dummy_scans

Module qc_report_wf has no input called inputnode.motion_file

Module qc_report_wf has no input called inputnode.temporal_mask

Module qc_report_wf has no input called inputnode.denoised_interpolated_bold

Module qc_report_wf has no input called inputnode.censored_denoised_bold

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/managers.py", line 599, in _run_server
    server.serve_forever()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/managers.py", line 184, in serve_forever
    sys.exit(0)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/util.py", line 133, in _remove_temp_dir
    rmtree(tempdir)
  File "/usr/local/miniconda/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/local/miniconda/lib/python3.10/shutil.py", line 681, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/local/miniconda/lib/python3.10/shutil.py", line 679, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs00000000023c80200011c27d'
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.10/multiprocessing/util.py", line 133, in _remove_temp_dir
    rmtree(tempdir)
  File "/usr/local/miniconda/lib/python3.10/shutil.py", line 731, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/usr/local/miniconda/lib/python3.10/shutil.py", line 729, in rmtree
    os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/pymp-hoin0vqz'
tsalo commented 1 week ago

I think I see the problem now. In the scenario where linc-qc and abcc-qc are disabled and CIFTI processing is enabled, it looks like init_qc_report_wf won't actually do anything. I'll push a fix ASAP.