search

bids-apps / CPAC

BIDS Application for the Configurable Pipeline for the Analysis of Connectomes (C-PAC)
Apache License 2.0
14 stars 18 forks source link

BIDS: session autoset to "1" #20

Closed jdkent closed 6 years ago

jdkent commented 6 years ago

Thanks for adding this BIDS-App, we are excited to get it working.

BACKGROUND I'm trying to get this working on our cluster which has singularity (v2.2.1) installed. I've built the singularity image today (after 3/4 of the commits applied today to this repository), and it appears to start running okay, but after an hour or so errors out.

QUESTION Why does CPAC think my subject label is fmriprep+controlGE140 when it's actually controlGE140 and my session label is 1 when it's actually "pre" or "post"?

This is the test-job script for our SGE cluster:

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M james-kent@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/PACR-AD/Imaging/BIDS/derivatives/code/cpac/out
#$ -e /Shared/vosslabhpc/Projects/PACR-AD/Imaging/BIDS/derivatives/code/cpac/err

singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/bids_cpac-2017-12-06-f45fd0b5142f.img \
/mnt/Projects/PACR-AD/Imaging/BIDS /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac \
participant --n_cpus 16 --mem_gb 32 --save_working_dir \
--participant_label controlGE140

This is stdout (minus the bids-validation):

#### Running C-PAC on ['controlGE140']
Number of participants to run in parallel: 1
Input directory: /mnt/Projects/PACR-AD/Imaging/BIDS
Output directory: /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac/output
Working directory: /scratch/working
Crash directory: /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac/crash
Log directory: /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac/log
Remove working directory: True
Available memory: 32.0 (GB)
Available threads: 16
Number of threads for ANTs: 4
sub-fmriprep+controlGE140 ses-1 is missing either an anat or rest (or both)

This is the stderr:

Traceback (most recent call last):
  File "/code/run.py", line 280, in <module>
    import CPAC
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/CPAC/__init__.py", line 21, in <module>
    import anat_preproc, \
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/CPAC/anat_preproc/__init__.py", line 1, in <module>
    from anat_preproc import create_anat_preproc
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/CPAC/anat_preproc/anat_preproc.py", line 1, in <module>
    from nipype.interfaces.afni import preprocess
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/__init__.py", line 49, in <module>
    from .pipeline import Node, MapNode, JoinNode, Workflow
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/__init__.py", line 10, in <module>
    from .engine import Node, MapNode, JoinNode, Workflow
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/__init__.py", line 12, in <module>
    from .workflows import Workflow
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.py", line 41, in <module>
    from ...interfaces.base import (traits, InputMultiPath, CommandLine,
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/interfaces/__init__.py", line 12, in <module>
    from .io import DataGrabber, DataSink, SelectFiles
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/interfaces/io.py", line 38, in <module>
    from ..utils.filemanip import copyfile, list_to_filename, filename_to_list
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/utils/filemanip.py", line 266, in <module>
    _cifs_table = _generate_cifs_table()
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/utils/filemanip.py", line 259, in _generate_cifs_table
    reverse=True)
  File "/usr/local/bin/miniconda/lib/python2.7/site-packages/nipype/utils/filemanip.py", line 258, in <lambda>
    key=lambda x: len(x[0]),
IndexError: list index out of range

This is data config yml that was generated

- anat: /mnt/Projects/PACR-AD/Imaging/BIDS/sub-controlGE140/ses-pre/anat/sub-controlGE140_ses-pre_T1w.nii.gz
  creds_path: null
  rest: {task-flanker: /mnt/Projects/PACR-AD/Imaging/BIDS/sub-controlGE140/ses-pre/func/sub-controlGE140_ses-pre_task-flanker_bold.nii.gz,
    task-rest: /mnt/Projects/PACR-AD/Imaging/BIDS/sub-controlGE140/ses-pre/func/sub-controlGE140_ses-pre_task-rest_bold.nii.gz}
  site_id: site-none
  subject_id: sub-controlGE140
  unique_id: ses-pre
- anat: /mnt/Projects/PACR-AD/Imaging/BIDS/sub-controlGE140/ses-post/anat/sub-controlGE140_ses-post_T1w.nii.gz
  creds_path: null
  rest: {task-flanker: /mnt/Projects/PACR-AD/Imaging/BIDS/sub-controlGE140/ses-post/func/sub-controlGE140_ses-post_task-flanker_bold.nii.gz,
    task-rest: /mnt/Projects/PACR-AD/Imaging/BIDS/sub-controlGE140/ses-post/func/sub-controlGE140_ses-post_task-rest_bold.nii.gz}
  site_id: site-none
  subject_id: sub-controlGE140
  unique_id: ses-post

This is the pipeline config that was generated:

FSLDIR: /usr/share/fsl/5.0
PRIORS_CSF: $priors_path/avg152T1_csf_bin.nii.gz
PRIORS_GRAY: $priors_path/avg152T1_gray_bin.nii.gz
PRIORS_WHITE: $priors_path/avg152T1_white_bin.nii.gz
Regressors:
- {compcor: 1, csf: 1, global: 1, gm: 0, linear: 1, motion: 1, pc1: 0, quadratic: 1,
  wm: 0}
- {compcor: 1, csf: 1, global: 0, gm: 0, linear: 1, motion: 1, pc1: 0, quadratic: 1,
  wm: 0}
TR: None
already_skullstripped: [0]
awsOutputBucketCredentials: null
boundaryBasedRegistrationSchedule: /usr/share/fsl/5.0/etc/flirtsch/bbr.sch
clusterSize: 27
configFileTwomm: $FSLDIR/etc/flirtsch/T1_2_MNI152_2mm.cnf
crashLogDirectory: /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac/crash
degCorrelationThreshold: 0.001
degCorrelationThresholdOption: [Sparsity threshold]
degWeightOptions: [true, true]
dilated_symmetric_brain_mask: $FSLDIR/data/standard/MNI152_T1_${resolution_for_anat}_brain_mask_symmetric_dil.nii.gz
eigCorrelationThreshold: 0.001
eigCorrelationThresholdOption: [Sparsity threshold]
eigWeightOptions: [false, true]
fdCalc: [Jenkinson]
fnirtConfig: T1_2_MNI152_2mm
func_reg_input: [Mean Functional]
func_reg_input_volume: 0
functionalMasking: [3dAutoMask]
fwhm: [6]
highPassFreqALFF: [0.01]
identityMatrix: /usr/share/fsl/5.0/etc/flirtsch/ident.mat
lateral_ventricles_mask: /usr/share/fsl/5.0/data/atlases/HarvardOxford/HarvardOxford-lateral-ventricles-thr25-2mm.nii.gz
lfcdCorrelationThreshold: 0.6
lfcdCorrelationThresholdOption: [Correlation threshold]
lfcdWeightOptions: [true, true]
logDirectory: /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac/log
lowPassFreqALFF: [0.1]
maxCoresPerParticipant: 16
maximumMemoryPerParticipant: 32.0
memoryAllocatedForDegreeCentrality: 1.0
modelConfigs: []
mrsNorm: true
nComponents: [5]
nuisanceBandpassFreq:
- [0.01, 0.1]
numGPAModelsAtOnce: 1
numParticipantsAtOnce: 1
numRemovePrecedingFrames: 1
numRemoveSubsequentFrames: 2
num_ants_threads: 4
outputDirectory: /mnt/Projects/PACR-AD/Imaging/BIDS/derivatives/cpac/output
parallelEnvironment: mpi_smp
pipelineName: analysis
priors_path: /usr/share/fsl/5.0/data/standard/tissuepriors/2mm
queue: all.q
reGenerateOutputs: false
ref_mask: /usr/share/fsl/5.0/data/standard/MNI152_T1_${resolution_for_anat}_brain_mask_symmetric_dil.nii.gz
regOption: [ANTS]
regWithSkull: [1]
removeWorkingDir: true
resolution_for_anat: 2mm
resolution_for_func_derivative: 3mm
resolution_for_func_preproc: 3mm
resourceManager: SGE
roiTSOutputs: [true, true]
runALFF: [1]
runBBReg: [1]
runFrequencyFiltering: [1, 0]
runFristonModel: [1]
runMedianAngleCorrection: [0]
runMotionSpike: ['Off']
runNetworkCentrality: [1]
runNuisance: [1]
runOnGrid: false
runROITimeseries: [1]
runReHo: [1]
runRegisterFuncToAnat: [1]
runRegisterFuncToMNI: [1]
runSCA: [1]
runScrubbing: [0]
runSegmentationPreprocessing: [1]
runSymbolicLinks: [0]
runVMHC: [1]
runZScoring: [0]
s3Encryption: [0]
sca_roi_paths:
- {/cpac_resources/cpac_templates/PNAS_Smith09_rsn10.nii.gz: DualReg}
scrubbingThreshold: [0.2]
slice_timing_correction: [1]
slice_timing_pattern: [Use NIFTI Header]
spikeThreshold: ['0.5']
startIdx: 4
stopIdx: None
targetAngleDeg: [90]
templateSpecificationFile: /cpac_resources/cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz
template_brain_only_for_anat: /usr/share/fsl/5.0/data/standard/MNI152_T1_${resolution_for_anat}_brain.nii.gz
template_brain_only_for_func: /usr/share/fsl/5.0/data/standard/MNI152_T1_${resolution_for_func_preproc}_brain.nii.gz
template_skull_for_anat: /usr/share/fsl/5.0/data/standard/MNI152_T1_${resolution_for_anat}.nii.gz
template_skull_for_func: /usr/share/fsl/5.0/data/standard/MNI152_T1_${resolution_for_func_preproc}.nii.gz
template_symmetric_brain_only: $FSLDIR/data/standard/MNI152_T1_${resolution_for_anat}_brain_symmetric.nii.gz
template_symmetric_skull: $FSLDIR/data/standard/MNI152_T1_${resolution_for_anat}_symmetric.nii.gz
tsa_roi_paths:
- {/cpac_resources/cpac_templates/CC200.nii.gz: Avg, /cpac_resources/cpac_templates/CC400.nii.gz: Avg,
  /cpac_resources/cpac_templates/PNAS_Smith09_rsn10.nii.gz: SpatialReg, /cpac_resources/cpac_templates/aal_mask_pad.nii.gz: Avg,
  /cpac_resources/cpac_templates/ez_mask_pad.nii.gz: Avg, /cpac_resources/cpac_templates/ho_mask_pad.nii.gz: Avg,
  /cpac_resources/cpac_templates/rois_3mm.nii.gz: Avg, /cpac_resources/cpac_templates/tt_mask_pad.nii.gz: Avg}
workingDirectory: /scratch/working

and this is how the example subject is organized:

sub-controlGE140
├── ses-post
│   ├── anat
│   │   ├── sub-controlGE140_ses-post_acq-1_T2w.nii.gz
│   │   ├── sub-controlGE140_ses-post_acq-2_T2w.nii.gz
│   │   └── sub-controlGE140_ses-post_T1w.nii.gz
│   ├── cbf
│   │   ├── sub-controlGE140_ses-post_acq-ASL_cbf.nii.gz
│   │   └── sub-controlGE140_ses-post_acq-cbf_cbf.nii.gz
│   ├── dwi
│   │   ├── sub-controlGE140_ses-post_acq-60D_dwi.bval
│   │   ├── sub-controlGE140_ses-post_acq-60D_dwi.bvec
│   │   ├── sub-controlGE140_ses-post_acq-60D_dwi.nii.gz
│   │   ├── sub-controlGE140_ses-post_acq-B0_dwi.bval
│   │   ├── sub-controlGE140_ses-post_acq-B0_dwi.bvec
│   │   └── sub-controlGE140_ses-post_acq-B0_dwi.nii.gz
│   ├── fmap
│   │   ├── sub-controlGE140_ses-post_fieldmap.json
│   │   ├── sub-controlGE140_ses-post_fieldmap.nii.gz
│   │   └── sub-controlGE140_ses-post_magnitude.nii.gz
│   └── func
│       ├── sub-controlGE140_ses-post_task-flanker_bold.nii.gz
│       ├── sub-controlGE140_ses-post_task-flanker_events.tsv
│       └── sub-controlGE140_ses-post_task-rest_bold.nii.gz
└── ses-pre
    ├── anat
    │   ├── sub-controlGE140_ses-pre_acq-1_T2w.nii.gz
    │   ├── sub-controlGE140_ses-pre_acq-2_T2w.nii.gz
    │   └── sub-controlGE140_ses-pre_T1w.nii.gz
    ├── cbf
    │   ├── sub-controlGE140_ses-pre_acq-ASL_cbf.nii.gz
    │   └── sub-controlGE140_ses-pre_acq-cbf_cbf.nii.gz
    ├── dwi
    │   ├── sub-controlGE140_ses-pre_acq-60D_dwi.bval
    │   ├── sub-controlGE140_ses-pre_acq-60D_dwi.bvec
    │   ├── sub-controlGE140_ses-pre_acq-60D_dwi.bvece
    │   ├── sub-controlGE140_ses-pre_acq-60D_dwi.nii.gz
    │   ├── sub-controlGE140_ses-pre_acq-B0_dwi.bval
    │   ├── sub-controlGE140_ses-pre_acq-B0_dwi.bvec
    │   └── sub-controlGE140_ses-pre_acq-B0_dwi.nii.gz
    ├── fmap
    │   ├── sub-controlGE140_ses-pre_fieldmap.json
    │   ├── sub-controlGE140_ses-pre_fieldmap.nii.gz
    │   └── sub-controlGE140_ses-pre_magnitude.nii.gz
    └── func
        ├── outpt.txt
        ├── sub-controlGE140_ses-pre_task-flanker_bold.nii.gz
        ├── sub-controlGE140_ses-pre_task-flanker_events.tsv
        └── sub-controlGE140_ses-pre_task-rest_bold.nii.gz

12 directories, 36 files

There are a couple top level json files too in the BIDS directory: fieldmap.json task-flanker_bold.json task-rest_bold.json

POSSIBLY RELEVANT I see that the session variable gets defined in bids_utils.py, but I can't readily parse how f_dict you pull bids session information.

ccraddock commented 6 years ago

At first glance, your data configuration file appears to be correct and appears to match the file structure you listed. To quickly make BIDS work with CPAC, we store the session information in the "unique_id" field of the data configuration dictionary. I am extremely confused about the '+fmriprep' being added to your subject name. I think that it is something to do with the '--participant_label' flag, which may be one of the least tested aspects of that code. Would you try using '--participant_index' instead with either 0 or 1?

BTW, I know that it is not well advertised, but you can download the latest version of the cpac bids app singularity image from here: https://fcp-indi.s3.amazonaws.com/resources/singularity_images/bids_cpac/bids_cpac_latest.img

ccraddock commented 6 years ago

on further thought, i don't know how the --participant_label flag could result in the error you are seeing, i will check it more in depth and see if I can figure out what is happening.

jdkent commented 6 years ago

Thank you! I tried the singularity container you provided and got further in the analysis, when I find time I will dig into what the differences between our singularity containers are and hopefully pin down this issue. Now I'm having another issue, but it's much more tractable. I'll open that one separately.