search

MICA-MNI / micapipe

micapipe from the Multimodal imaging and connectome analysis lab (http://mica-mni.github.io) at the Montreal Neurological Institute. Read The Docs documentation below
http://micapipe.readthedocs.io
GNU General Public License v3.0
86 stars 29 forks source link

QC_subj terminates early without all outputs #93

Closed araikes closed 1 year ago

araikes commented 1 year ago

Sorry... I was hoping I was done running into issues/questions.

Relative other issues #89, #92, my processing finished for -proc_structural, -proc_surf, -post_structural, and -proc_dwi with all outputs being produced. I went to run -QC_subj and get the following output/error. Only the structural PDF is produced and even then, it terminates at the white matter partial volume images. I don't get any output for the surface pipeline, post-structural, or dwi.

Also... was the TDI file removed at some point? There's still a reference to it in QC.sh but not in proc_dwi

[adamraikes@r5u13n1 allo]$ singularity run --nv --containall -B $PWD/derivatives/fastsurfer:$PWD/derivatives/fastsurfer -B $PWD/nifti:$PWD/nifti:ro -B $PWD/derivatives/micapipe_0.2.2:$PWD/derivatives/micapipe_0.2.2 -B /tmp:/tmp -B /groups/adamraikes/license.txt:/opt/freesurfer-7.3.2/license.txt $MICA -sub allo102 -ses bline -bids $PWD/nifti -out $PWD/derivatives/micapipe_0.2.2 -QC_subj -fs_licence /opt/freesurfer-7.3.2/license.txt                                                                                                                                                                                            
INFO:    underlay of /usr/bin/nvidia-smi required more than 50 (875) bind mounts
Some packages in this Docker container are non-free
If you are considering commercial use of this container, please consult the relevant license:
https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Licence
INFO: /home/u26/adamraikes/matlab/startup.m does not exist ... creating

micapipe -sub allo102 -ses bline -bids /xdisk/adamraikes/allo/nifti -out /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2 -QC_subj -fs_licence /opt/freesurfer-7.3.2/license.txt

-------------------------------------------------------------
    MICA pipeline - (Version v0.2.2 'Northern flicker') 
        Subject: allo102 Session: ses-bline
-------------------------------------------------------------
        Freesurfer licence: /opt/freesurfer-7.3.2/license.txt

-------------------------------------------------------------
    MICAPIPE: Creating a QC report for sub-allo102_ses-bline
-------------------------------------------------------------
        Modules processed:  4
        sub:    allo102
        out:    /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0
        bids:   /xdisk/adamraikes/allo/nifti
        ses:    ses-bline
        PROC:   container_micapipe-v0.2.2
        MICAPIPE:   /opt/micapipe
        conda     /opt/miniconda-22.11.1/envs/micapipe
        tmpDir  /tmp

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:58:53 MST 2023:
COMMAND -->   mkdir -p /tmp/9948_micapipe_QC_allo102  

-------------------------------------------------------------
    Generating necessary files for QC report
-------------------------------------------------------------

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:58:53 MST 2023:
COMMAND -->   mrconvert /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_5tt.nii.gz -coord 3 0 -axes 0,1,2 /tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_5tt.nii.gz -force  
mrconvert: [WARNING] existing output files will be overwritten
mrconvert: [100%] uncompressing image "/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_5tt.nii.gz"
mrconvert: [100%] copying from "/xdisk/ada...ce-nativepro_T1w_5tt.nii.gz" to "/tmp/9948_...ivepro_T1w_brain_5tt.nii.gz"
mrconvert: [100%] compressing image "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_5tt.nii.gz"

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:58:55 MST 2023:
COMMAND -->   antsApplyTransforms -d 3 -v -u int -o /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-MNI152_2_T1w_brain.nii.gz -i /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_brain.nii.gz -r /opt/micapipe/MNI152Volumes/MNI152_T1_2mm_brain.nii.gz -t /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_2mm_mode-image_desc-SyN_1Warp.nii.gz -t /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_2mm_mode-image_desc-SyN_0GenericAffine.mat  
Using double precision for computations.
Input scalar image: /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_brain.nii.gz
Reference image: /opt/micapipe/MNI152Volumes/MNI152_T1_2mm_brain.nii.gz
=============================================================================
The composite transform comprises the following transforms (in order): 
  1. /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_2mm_mode-image_desc-SyN_0GenericAffine.mat (type = AffineTransform)
  2. /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_2mm_mode-image_desc-SyN_1Warp.nii.gz (type = DisplacementFieldTransform)
=============================================================================
Default pixel value: 0
Interpolation type: LinearInterpolateImageFunction
Output warped image: /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-MNI152_2_T1w_brain.nii.gz

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:58:55 MST 2023:
COMMAND -->   antsApplyTransforms -d 3 -v -u int -o /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-MNI152_0.8_T1w_brain.nii.gz -i /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_brain.nii.gz -r /opt/micapipe/MNI152Volumes/MNI152_T1_0.8mm_brain.nii.gz -t /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_0.8mm_mode-image_desc-SyN_1Warp.nii.gz -t /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_0.8mm_mode-image_desc-SyN_0GenericAffine.mat  
Using double precision for computations.
Input scalar image: /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_brain.nii.gz
Reference image: /opt/micapipe/MNI152Volumes/MNI152_T1_0.8mm_brain.nii.gz
=============================================================================
The composite transform comprises the following transforms (in order): 
  1. /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_0.8mm_mode-image_desc-SyN_0GenericAffine.mat (type = AffineTransform)
  2. /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/xfm/sub-allo102_ses-bline_from-nativepro_brain_to-MNI152_0.8mm_mode-image_desc-SyN_1Warp.nii.gz (type = DisplacementFieldTransform)
=============================================================================
Default pixel value: 0
Interpolation type: LinearInterpolateImageFunction
Output warped image: /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-MNI152_0.8_T1w_brain.nii.gz
ls: cannot access '/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/QC/sub-allo102_ses-bline_module-proc_func-*.json': No such file or directory

[ INFO ]..... proc_dwi 

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:58:59 MST 2023:
COMMAND -->   fslmaths /xdisk/adamraikes/allo/nifti/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_acq-b1000_dir-PA_run-001_dwi.nii.gz -Tmean /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_acq-b1000_dir-PA_run-001_dwi_mean.nii.gz  

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:59:03 MST 2023:
COMMAND -->   fslmaths /xdisk/adamraikes/allo/nifti/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_acq-b2000_dir-PA_run-001_dwi.nii.gz -Tmean /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_acq-b2000_dir-PA_run-001_dwi_mean.nii.gz  

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:59:08 MST 2023:
COMMAND -->   mrmath /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_space-dwi_desc-preproc_dwi.mif mean /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-dwi_desc-preproc_dwi_mean.nii.gz -axis 3 -nthreads 6  
mrmath: [100%] preloading data for "/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_space-dwi_desc-preproc_dwi.mif"
mrmath: [100%] computing mean along axis 3...
mrmath: [100%] compressing image "/tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-dwi_desc-preproc_dwi_mean.nii.gz"

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:59:09 MST 2023:
COMMAND -->   mrconvert /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_space-dwi_model-CSD_map-FOD_desc-wmNorm.nii.gz -coord 3 0 -axes 0,1,2 /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-dwi_model-CSD_map-FOD_desc-wmNorm.nii.gz -force -nthreads 6  
mrconvert: [WARNING] existing output files will be overwritten
mrconvert: [100%] uncompressing image "/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_space-dwi_model-CSD_map-FOD_desc-wmNorm.nii.gz"
mrconvert: [100%] copying from "/xdisk/ada..._map-FOD_desc-wmNorm.nii.gz" to "/tmp/9948_..._map-FOD_desc-wmNorm.nii.gz"
mrconvert: [100%] compressing image "/tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-dwi_model-CSD_map-FOD_desc-wmNorm.nii.gz"

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:59:11 MST 2023:
COMMAND -->   mrconvert /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_space-dwi_desc-5tt.nii.gz -coord 3 0 -axes 0,1,2 /tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-dwi_desc-5tt.nii.gz -force -nthreads 6  
mrconvert: [WARNING] existing output files will be overwritten
mrconvert: [100%] uncompressing image "/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/dwi/sub-allo102_ses-bline_space-dwi_desc-5tt.nii.gz"
mrconvert: [100%] copying from "/xdisk/ada...e_space-dwi_desc-5tt.nii.gz" to "/tmp/9948_...e_space-dwi_desc-5tt.nii.gz"
mrconvert: [100%] compressing image "/tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-dwi_desc-5tt.nii.gz"

-------------------------------------------------------------
    Generating QC report
-------------------------------------------------------------

adamraikes @ r5u13n1.puma.hpc.arizona.edu Wed Aug  9 13:59:12 MST 2023:
COMMAND -->   python /opt/micapipe/functions/QC.py -sub sub-allo102 -out /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0 -bids /xdisk/adamraikes/allo/nifti -ses bline -tmpDir /tmp/9948_micapipe_QC_allo102 -micapipe /opt/micapipe  

Check inputs:
  -img   : "['/xdisk/adamraikes/allo/nifti/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_run-001_T1w.nii.gz']"
   Number of img: "1"
  -out   : "/tmp/9948_micapipe_QC_allo102/t1w_mainScan1.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating single image png

Check inputs:
  -img   : "['/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w.nii.gz']"
   Number of img: "1"
  -out   : "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_screenshot.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating single image png

Check inputs:
  -img   : "['/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w.nii.gz', '/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_brain_mask.nii.gz']"
   Number of img: "2"
  -out   : "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_mask_screenshot.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating image overlaying NIFTI png

Check inputs:
  -img   : "['/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w.nii.gz', '/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_5tt.nii.gz']"
   Number of img: "2"
  -out   : "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_5tt_screenshot.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating image overlaying NIFTI png

Check inputs:
  -img   : "['/tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-MNI152_0.8_T1w_brain.nii.gz', '/opt/micapipe/MNI152Volumes/MNI152_T1_0.8mm_brain_mask.nii.gz']"
   Number of img: "2"
  -out   : "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_mni152_08_screenshot.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating image overlaying NIFTI png

Check inputs:
  -img   : "['/tmp/9948_micapipe_QC_allo102/sub-allo102_ses-bline_space-MNI152_2_T1w_brain.nii.gz', '/opt/micapipe/MNI152Volumes/MNI152_T1_2mm_brain_mask.nii.gz']"
   Number of img: "2"
  -out   : "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_mni152_2_screenshot.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating image overlaying NIFTI png

Check inputs:
  -img   : "['/xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/anat/sub-allo102_ses-bline_space-nativepro_T1w_brain_pve_2.nii.gz']"
   Number of img: "1"
  -out   : "/tmp/9948_micapipe_QC_allo102/nativepro_T1w_brain_pve_2_screenshot.png"
  -roi   : "False"
  -title :  ""

[INFO]... Creating single image png
bline
Traceback (most recent call last):
  File "/opt/micapipe/functions/QC.py", line 1695, in <module>
    static_report = qc_module_function['functions'][i](j)
  File "/opt/micapipe/functions/QC.py", line 482, in qc_proc_surf
    display.start()
  File "/opt/miniconda-22.11.1/envs/micapipe/lib/python3.9/site-packages/pyvirtualdisplay/display.py", line 72, in start
    self._obj.start()
  File "/opt/miniconda-22.11.1/envs/micapipe/lib/python3.9/site-packages/pyvirtualdisplay/abstractdisplay.py", line 149, in start
    self._start1_has_displayfd()
  File "/opt/miniconda-22.11.1/envs/micapipe/lib/python3.9/site-packages/pyvirtualdisplay/abstractdisplay.py", line 197, in _start1_has_displayfd
    self.display = int(self._wait_for_pipe_text(rfd))
  File "/opt/miniconda-22.11.1/envs/micapipe/lib/python3.9/site-packages/pyvirtualdisplay/abstractdisplay.py", line 297, in _wait_for_pipe_text
    raise XStartError(
pyvirtualdisplay.abstractdisplay.XStartError: Xvfb program closed. command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '900x250x24', '-displayfd', '5'] stderr: b'_XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root\nThe XKEYBOARD keymap compiler (xkbcomp) reports:\n> Error:            Cannot open "/var/lib/xkb/server-0.xkm" to write keyboard description\n>                   Exiting\nThe XKEYBOARD keymap compiler (xkbcomp) reports:\n> Error:            Cannot open "/var/lib/xkb/server-0.xkm" to write keyboard description\n>                   Exiting\nXKB: Failed to compile keymap\nKeyboard initialization failed. This could be a missing or incorrect setup of xkeyboard-config.\n(EE) \nFatal server error:\n(EE) Failed to activate virtual core keyboard: 2(EE) \n'

[ INFO ]..... Outputs: 
 /xdisk/adamraikes/allo/derivatives/micapipe_0.2.2/micapipe_v0.2.0/sub-allo102/ses-bline/QC/sub-allo102_ses-bline_module-proc_structural_qc-report.pdf 

-------------------------------------------------------------
    QC html creation ended in  1.033 minutes :
        Output file path: 
-------------------------------------------------------------
rcruces commented 1 year ago

It appears that there is an error with the permissions when the virtual display is creating a temporary file Owner of /tmp/.X11-unix should be set to root. I don't know what are the permissions of your /tmp directory. This issue is only affecting the plots over the surface that require the virtual display. we test it in a shared /tmp directory with the following permissions and it worked:

48 drwxrwxrwt 23 root    root 45056 Aug  9 21:27 .
 4 drwxr-xr-x 64 root    root  4096 Jun 10  2022 ..

I haven't seen this before, but the virtual display writes a file under $tmp called .X11-unix, which seems to be necessary for the surface plots, if that file does not have the permissions it seems to fail to generate the plots. My guess is that this directory is inheriting permissions from /tmp. and it might not be able to read it. You could set your tmp dir to another place with more open permissions and then erase it at the end of the processing.

araikes commented 1 year ago

Fixed... may want to add to the documentation for singularity.

Use the --writable-tmpfs flag. As in #91, I'm using --containall so I'm also explicitly binding /tmp.

singularity run --writable-tmpfs --containall -B /tmp:/tmp ...... etc

rcruces commented 1 year ago

This was added to the latest documentation. Thank you