search

bids-apps / MRtrix3_connectome

Generate subject connectomes from raw BIDS data & perform inter-subject connection density normalisation, using the MRtrix3 software package.
http://www.mrtrix.org/
Apache License 2.0
49 stars 26 forks source link

Preprocessed T1w data not found at participant level #100

Closed treanus closed 3 years ago

treanus commented 3 years ago

I ran the preproc level. When running the participant level the preprocessed T1w data are not found at the 5ttgen command.

Seems that https://github.com/BIDS-Apps/MRtrix3_connectome/blob/2f4f793a5d339f0abc03a42bc0269f4605a1b0b8/mrtrix3_connectome.py#L2172 is not fiding the correct image.

The scratch dir does not contain T1.mif, see:

(base) stefan@broca:/DATA/COVID_DTI/mrtrix3_connectome.py-tmp-DA4QTG$ tree
.
├── dwi_mask.mif
├── dwi.mif
├── error.txt
├── FOD_WM.mif
├── log.txt
├── response_csf.txt
├── response_gm.txt
├── response_wm.txt
├── T1.json
├── T1_masked.mif
└── T1_mask.mif

0 directories, 11 files

The output of mrtrix3_connectome is:

docker run -i --rm -v /DATA/COVID_DTI/BIDS:/bids_dataset -v /DATA/COVID_DTI/MRtrix3_connectome:/output --gpus all bids/mrtrix3_connectome /bids_dataset /output participant --participant_label iz02 --session_label 02 --output_verbosity 4 --template_reg ants --parcellation desikan --n_cpus 15
mrtrix3_connectome.py: 
mrtrix3_connectome.py: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
mrtrix3_connectome.py: 
Command:  bids-validator /bids_dataset
mrtrix3_connectome.py: 30 total sessions found in directory '/bids_dataset'; 1 will be processed
mrtrix3_connectome.py: [WARNING] Volumetric template registration not being performed; --template_reg option ignored
mrtrix3_connectome.py: Commencing execution for session: 'sub-iz02_ses-02'
mrtrix3_connectome.py: Generated scratch directory: /mrtrix3_connectome.py-tmp-TMZIIE/
mrtrix3_connectome.py: Importing pre-processed data into scratch directory
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-iz02/ses-02/dwi/sub-iz02_ses-02_desc-preproc_dwi.nii.gz /mrtrix3_connectome.py-tmp-TMZIIE/dwi.mif -fslgrad /output/MRtrix3_connectome-preproc/sub-iz02/ses-02/dwi/sub-iz02_ses-02_desc-preproc_dwi.bvec /output/MRtrix3_connectome-preproc/sub-iz02/ses-02/dwi/sub-iz02_ses-02_desc-preproc_dwi.bval -strides 0,0,0,1
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-iz02/ses-02/dwi/sub-iz02_ses-02_desc-brain_mask.nii.gz /mrtrix3_connectome.py-tmp-TMZIIE/dwi_mask.mif -datatype bit
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-iz02/ses-02/anat/sub-iz02_ses-02_desc-preproc_T1w.nii.gz /mrtrix3_connectome.py-tmp-TMZIIE/T1_masked.mif
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-iz02/ses-02/anat/sub-iz02_ses-02_desc-brain_mask.nii.gz /mrtrix3_connectome.py-tmp-TMZIIE/T1_mask.mif -datatype bit
mrtrix3_connectome.py: Changing to scratch directory (/mrtrix3_connectome.py-tmp-TMZIIE/)
mrtrix3_connectome.py: Estimating tissue response functions for spherical deconvolution
Command:  dwi2response dhollander dwi.mif response_wm.txt response_gm.txt response_csf.txt -mask dwi_mask.mif
mrtrix3_connectome.py: Estimating Fibre Orientation Distribution image
Command:  dwi2fod msmt_csd dwi.mif response_wm.txt FOD_WM.mif response_csf.txt FOD_CSF.mif -lmax 10,0
mrtrix3_connectome.py: Generating five-tissue-type (5TT) image for Anatomically-Constrained Tractography (ACT)
Command:  5ttgen fsl T1.mif 5TT.mif -mask T1_mask.mif

mrtrix3_connectome.py: [ERROR] 5ttgen fsl T1.mif 5TT.mif -mask T1_mask.mif (mrtrix3_connectome.py:2176)
mrtrix3_connectome.py: [ERROR] Information from failed command:
mrtrix3_connectome.py:
                       5ttgen: 
                       5ttgen: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
                       5ttgen: 
                       5ttgen: Generated scratch directory: /mrtrix3_connectome.py-tmp-TMZIIE/5ttgen-tmp-7M9J3U/
                       mrinfo: [ERROR] failed to open key/value file "/mrtrix3_connectome.py-tmp-TMZIIE/T1.mif": No such file or directory
                       mrinfo: [ERROR] error opening image "/mrtrix3_connectome.py-tmp-TMZIIE/T1.mif"

                       5ttgen: [ERROR] Could not access header information for image '/mrtrix3_connectome.py-tmp-TMZIIE/T1.mif'
mrtrix3_connectome.py:
mrtrix3_connectome.py: [ERROR] For debugging, inspect contents of scratch directory: /mrtrix3_connectome.py-tmp-TMZIIE/
mrtrix3_connectome.py: Scratch directory retained; location: /mrtrix3_connectome.py-tmp-TMZIIE/
treanus commented 3 years ago

I did some debugging and may have found a cause.

I think this was related to the preproc level that I run previously. I ran that using the command line mrtrix3_connectome.py. Without having ROBEX installed. In that case fsl_anat is used, and the T1.mif is not generated. Installing ROBEX and rerunning the preproc level, solves this.

Best, Stefan

treanus commented 3 years ago

Having a similar problem at a later stage of processing when the tdi_t1.mif is being made. The preproc step did run fine.

docker run -i --rm -v /DATA/COVID_DTI/BIDS:/bids_dataset -v /DATA/COVID_DTI/MRtrix3_connectome:/output --gpus all bids/mrtrix3_connectome /bids_dataset /output participant --participant_label niz13 --output_verbosity 4 --template_reg ants --parcellation desikan --n_cpus 8
mrtrix3_connectome.py: 
mrtrix3_connectome.py: Note that this script makes use of commands / algorithms that have relevant articles for citation; INCLUDING FROM EXTERNAL SOFTWARE PACKAGES. Please consult the help page (-help option) for more information.
mrtrix3_connectome.py: 
Command:  bids-validator /bids_dataset
mrtrix3_connectome.py: 30 total sessions found in directory '/bids_dataset'; 1 will be processed
mrtrix3_connectome.py: [WARNING] Volumetric template registration not being performed; --template_reg option ignored
mrtrix3_connectome.py: Commencing execution for session: 'sub-niz13_ses-01'
mrtrix3_connectome.py: Generated scratch directory: /mrtrix3_connectome.py-tmp-DYPGLI/
mrtrix3_connectome.py: Importing pre-processed data into scratch directory
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-niz13/ses-01/dwi/sub-niz13_ses-01_desc-preproc_dwi.nii.gz /mrtrix3_connectome.py-tmp-DYPGLI/dwi.mif -fslgrad /output/MRtrix3_connectome-preproc/sub-niz13/ses-01/dwi/sub-niz13_ses-01_desc-preproc_dwi.bvec /output/MRtrix3_connectome-preproc/sub-niz13/ses-01/dwi/sub-niz13_ses-01_desc-preproc_dwi.bval -strides 0,0,0,1
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-niz13/ses-01/dwi/sub-niz13_ses-01_desc-brain_mask.nii.gz /mrtrix3_connectome.py-tmp-DYPGLI/dwi_mask.mif -datatype bit
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-niz13/ses-01/anat/sub-niz13_ses-01_desc-preproc_T1w.nii.gz /mrtrix3_connectome.py-tmp-DYPGLI/T1.mif
Command:  mrconvert /output/MRtrix3_connectome-preproc/sub-niz13/ses-01/anat/sub-niz13_ses-01_desc-brain_mask.nii.gz /mrtrix3_connectome.py-tmp-DYPGLI/T1_mask.mif -datatype bit
mrtrix3_connectome.py: Changing to scratch directory (/mrtrix3_connectome.py-tmp-DYPGLI/)
mrtrix3_connectome.py: Estimating tissue response functions for spherical deconvolution
Command:  dwi2response dhollander dwi.mif response_wm.txt response_gm.txt response_csf.txt -mask dwi_mask.mif
mrtrix3_connectome.py: Estimating Fibre Orientation Distribution image
Command:  dwi2fod msmt_csd dwi.mif response_wm.txt FOD_WM.mif response_csf.txt FOD_CSF.mif -lmax 10,0
mrtrix3_connectome.py: Generating five-tissue-type (5TT) image for Anatomically-Constrained Tractography (ACT)
Command:  5ttgen fsl T1.mif 5TT.mif -mask T1_mask.mif
Command:  5tt2vis 5TT.mif vis.mif
mrtrix3_connectome.py: Getting grey matter parcellation in subject space using FreeSurfer
Function: posix.symlink('/opt/freesurfer/subjects/fsaverage', 'fsaverage')
Function: posix.symlink('/opt/freesurfer/subjects/lh.EC_average', 'lh.EC_average')
Function: posix.symlink('/opt/freesurfer/subjects/rh.EC_average', 'rh.EC_average')
Command:  mrconvert T1.mif T1.nii -strides +1,+2,+3
Command:  recon-all -sd /mrtrix3_connectome.py-tmp-DYPGLI/ -subjid freesurfer -i T1.nii
Command:  recon-all -sd /mrtrix3_connectome.py-tmp-DYPGLI/ -subjid freesurfer -all -parallel -openmp 8
Command:  labelconvert freesurfer/mri/aparc+aseg.mgz /opt/freesurfer/FreeSurferColorLUT.txt /opt/mrtrix3/share/mrtrix3/labelconvert/fs_default.txt parc_init.mif
Command:  labelsgmfix parc_init.mif T1.nii /opt/mrtrix3/share/mrtrix3/labelconvert/fs_default.txt parc.mif
Command:  label2colour parc.mif parcRGB.mif -lut /opt/mrtrix3/share/mrtrix3/labelconvert/fs_default.txt
mrtrix3_connectome.py: Performing whole-brain fibre-tracking
Command:  tckgen FOD_WM.mif tractogram_3486000.tck -act 5TT.mif -backtrack -crop_at_gmwmi -maxlength 250 -power 0.33 -select 3486000 -seed_dynamic FOD_WM.mif
mrtrix3_connectome.py: Running the SIFT2 algorithm to assign weights to individual streamlines
Command:  tcksift2 tractogram_3486000.tck FOD_WM.mif weights.csv -act 5TT.mif -out_mu mu.txt -fd_scale_gm
Function: shutil.move('tractogram_3486000.tck', 'tractogram.tck')
mrtrix3_connectome.py: Producing Track Density Images (TDIs)
Command:  tckmap tractogram.tck - -tck_weights_in weights.csv -template FOD_WM.mif -precise | mrcalc - 0.000369871 -mult tdi_dwi.mif
Command:  tckmap tractogram.tck - -tck_weights_in weights.csv -template T1.mif -precise | mrcalc - 0.000369871 -mult tdi_T1.mif
Command:  tckmap tractogram.tck tdi_hires.mif -tck_weights_in weights.csv -vox 0.25 -datatype uint16
mrtrix3_connectome.py: Combining whole-brain tractogram with grey matter parcellation to produce the connectome
Command:  tck2connectome tractogram.tck parc.mif connectome.csv -tck_weights_in weights.csv -out_assignments assignments.csv
Command:  tck2connectome tractogram.tck parc.mif meanlength.csv -tck_weights_in weights.csv -scale_length -stat_edge mean
mrtrix3_connectome.py: Generating geometric data for enhanced connectome visualisation
Command:  connectome2tck tractogram.tck assignments.csv exemplars.tck -tck_weights_in weights.csv -exemplars parc.mif -files single
Command:  label2mesh parc.mif nodes.obj
Command:  meshfilter nodes.obj smooth nodes_smooth.obj
mrtrix3_connectome.py: Processing for session "sub-niz13_ses-01" completed; writing results to output directory
Function: os.makedirs('/output/MRtrix3_connectome-participant/sub-niz13/ses-01/anat')
Function: os.makedirs('/output/MRtrix3_connectome-participant/sub-niz13/ses-01/connectome')
Function: os.makedirs('/output/MRtrix3_connectome-participant/sub-niz13/ses-01/dwi')
Function: os.makedirs('/output/MRtrix3_connectome-participant/sub-niz13/ses-01/tractogram')
Command:  mrconvert T1.mif /output/MRtrix3_connectome-participant/sub-niz13/ses-01/anat/sub-niz13_ses-01_desc-preproc_T1w.nii.gz  -strides +1,+2,+3 -clear_property comments
Function: shutil.copyfile('meanlength.csv', '/output/MRtrix3_connectome-participant/sub-niz13/ses-01/connectome/sub-niz13_ses-01_desc-desikan_meanlength.csv')
Function: shutil.copyfile('T1.json', '/output/MRtrix3_connectome-participant/sub-niz13/ses-01/anat/sub-niz13_ses-01_desc-preproc_T1w.json')
Function: shutil.copyfile('connectome.csv', '/output/MRtrix3_connectome-participant/sub-niz13/ses-01/connectome/sub-niz13_ses-01_desc-desikan_connectome.csv')
Command:  mrconvert tdi_dwi.mif /output/MRtrix3_connectome-participant/sub-niz13/ses-01/tractogram/sub-niz13_ses-01_space-dwi_tdi.nii.gz -strides +1,+2,+3 -clear_property comments
Command:  mrconvert FOD_WM.mif /output/MRtrix3_connectome-participant/sub-niz13/ses-01/dwi/sub-niz13_ses-01_tissue-WM_ODF.nii.gz -strides +1,+2,+3,+4 -clear_property comments
Function: shutil.copyfile('5TT.json', '/output/MRtrix3_connectome-participant/sub-niz13/ses-01/anat/sub-niz13_ses-01_desc-5tt_probseg.json')
Command:  mrconvert tdi_t1.mif /output/MRtrix3_connectome-participant/sub-niz13/ses-01/tractogram/sub-niz13_ses-01_space-T1w_tdi.nii.gz -strides +1,+2,+3 -clear_property comments

mrtrix3_connectome.py: [ERROR] mrconvert tdi_t1.mif /output/MRtrix3_connectome-participant/sub-niz13/ses-01/tractogram/sub-niz13_ses-01_space-T1w_tdi.nii.gz -strides +1,+2,+3 -clear_property comments (mrtrix3_connectome.py:2781)
mrtrix3_connectome.py: [ERROR] Information from failed command:
mrtrix3_connectome.py:
                       mrconvert: [ERROR] failed to open key/value file "tdi_t1.mif": No such file or directory
                       mrconvert: [ERROR] error opening image "tdi_t1.mif"
mrtrix3_connectome.py:
mrtrix3_connectome.py: [ERROR] For debugging, inspect contents of scratch directory: /mrtrix3_connectome.py-tmp-DYPGLI/
mrtrix3_connectome.py: Scratch directory retained; location: /mrtrix3_connectome.py-tmp-DYPGLI/
Lestropie commented 3 years ago

Sorry for taking forever to respond; well and truly behind the 8-ball right now...

Looks like two independent problems, which just arise from not ensuring identical scratch file names as they are used in different locations. Ideally I'd have CI catching things like this... See #104.

Rob