search

HBClab / SoftwareIssues

Issues when running primarily fmriprep/mriqc/xcpengine (primarily on argon)
0 stars 0 forks source link

CREST fmriprep: PIL224 #13

Open ariveradompenciel opened 4 years ago

ariveradompenciel commented 4 years ago

Script/Job File

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M adriana-rivera-dompenciel@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg \
/mnt/Projects/CREST/ /mnt/Projects/CREST/derivatives \
participant --participant_label PIL224 \
-w /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

Email Message

Job 912198 (sub-PIL224.job) Complete
User             = ariveradompenciel
Queue            = UI@argon-itf-cf48-33.hpc
Host             = argon-itf-cf48-33.hpc
Start Time       = 05/20/2020 08:52:54
End Time         = 05/20/2020 09:16:41
User Time        = 00:01:16
System Time      = 00:00:03
Wallclock Time   = 00:23:47
CPU              = 01:55:43
Max vmem         = 5.365G
Exit Status      = 1

Output File

200520-09:13:31,60 nipype.workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_PIL224_wf.anat_preproc_wf.surface_recon_wf.autorecon1" (/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_PIL224_wf/anat_preproc_wf/surface_recon_wf/autorecon1)
200520-09:13:32,128 nipype.workflow ERROR:
     Node autorecon1 failed to run on host argon-itf-cf48-33.hpc.
200520-09:13:32,132 nipype.workflow ERROR:
     Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-PIL224/log/20200520-085300_dfc7d130-e992-4fe5-86a8-831e7bac0ef2/crash-20200520-091332-ariveradompenciel-autorecon1-a5d0efb8-9e4e-4c16-a510-eca681a59fc7.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 1038, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 975, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
recon-all -autorecon1 -i /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_PIL224_wf/anat_preproc_wf/anat_template_wf/t1_conform/mapflow/_t1_conform0/sub-PIL224_T1w_ras.nii.gz -noskullstrip -openmp 10 -subjid sub-PIL224 -sd /mnt/Projects/CREST/derivatives/freesurfer
Standard output:
Subject Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a
Current Stamp: freesurfer-Linux-centos6_x86_64-stable-pub-v6.0.1-f53a55a
INFO: SUBJECTS_DIR is /mnt/Projects/CREST/derivatives/freesurfer
Actual FREESURFER_HOME /opt/freesurfer
Linux argon-itf-cf48-33.hpc 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
'/opt/freesurfer/bin/recon-all' -> '/mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/scripts/recon-all.local-copy'
/mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224

 mri_convert /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_PIL224_wf/anat_preproc_wf/anat_template_wf/t1_conform/mapflow/_t1_conform0/sub-PIL224_T1w_ras.nii.gz /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig/001.mgz 

mri_convert.bin /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_PIL224_wf/anat_preproc_wf/anat_template_wf/t1_conform/mapflow/_t1_conform0/sub-PIL224_T1w_ras.nii.gz /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig/001.mgz 
$Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp $
reading from /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_PIL224_wf/anat_preproc_wf/anat_template_wf/t1_conform/mapflow/_t1_conform0/sub-PIL224_T1w_ras.nii.gz...
TR=2530.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (1, 0, 0)
j_ras = (0, 0.985109, 0.171929)
k_ras = (0, -0.171929, 0.985109)
writing to /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig/001.mgz...
#--------------------------------------------
#@# MotionCor Wed May 20 09:13:19 CDT 2020
Found 1 runs
/mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig/001.mgz
Checking for (invalid) multi-frame inputs...
WARNING: only one run found. This is OK, but motion
correction cannot be performed on one run, so I'll
copy the run to rawavg and continue.

 cp /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig/001.mgz /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/rawavg.mgz 

/mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224

 mri_convert /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/rawavg.mgz /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig.mgz --conform 

mri_convert.bin /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/rawavg.mgz /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig.mgz --conform 
WARNING ==================++++++++++++++++++++++++=======================================
The physical sizes are (256.00 mm, 336.00 mm, 256.00 mm), which cannot fit in 256^3 mm^3 volume.
The resulting volume will have 336 slices.
If you find problems, please let us know (freesurfer@nmr.mgh.harvard.edu).
==================================================++++++++++++++++++++++++===============

$Id: mri_convert.c,v 1.226 2016/02/26 16:15:24 mreuter Exp $
reading from /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/rawavg.mgz...
TR=2530.00, TE=0.00, TI=0.00, flip angle=0.00
i_ras = (1, 0, 0)
j_ras = (0, 0.985109, 0.171929)
k_ras = (0, -0.171929, 0.985109)
changing data type from float to uchar (noscale = 0)...
MRIchangeType: Building histogram 
Reslicing using trilinear interpolation 
writing to /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/mri/orig.mgz...

****************************************
ERROR! FOV=336.000 > 256
Include the flag -cw256 with recon-all!
Inspect orig.mgz to ensure the head is fully visible.
****************************************

Linux argon-itf-cf48-33.hpc 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-PIL224 exited with ERRORS at Wed May 20 09:13:30 CDT 2020

For more details, see the log file /mnt/Projects/CREST/derivatives/freesurfer/sub-PIL224/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Standard error:

Return code: 1

Error (stderr file)

/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:116: UserWarning: 'dataset_description.json' file is missing from project root. You may want to set the root path to a valid BIDS project.
  warnings.warn("'dataset_description.json' file is missing from "
/usr/local/miniconda/lib/python3.6/site-packages/nitime/utils.py:980: FutureWarning: Conversion of the second argument of issubdtype from `complex` to `np.complexfloating` is deprecated. In future, it will be treated as `np.complex128 == np.dtype(complex).type`.
  complex_result = (np.issubdtype(in1.dtype, np.complex) or
/usr/local/miniconda/lib/python3.6/site-packages/nitime/utils.py:981: FutureWarning: Conversion of the second argument of issubdtype from `complex` to `np.complexfloating` is deprecated. In future, it will be treated as `np.complex128 == np.dtype(complex).type`.
  np.issubdtype(in2.dtype, np.complex))
Errors occurred while generating reports for participants: PIL224 (1).
jdkent commented 4 years ago

if you have fsl installed, could you give the output of fslhd <name of anatomical file> or with fmriprep singularity image you can execute something like:

singularity exec -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL224/path/to/anatomical/file

and then do the same for a participant that has successfully completed in the PIL project (are there any?)

I want to know if this participant's anatomical image is a different size than another participant's anatomical image.

ariveradompenciel commented 4 years ago

do not have fsl. I was also unsure of which anat file to use, so tried three different files (two of which I saw in error output) for singularity. all three failed (as well as for a successfully run PIL sub):

command 1

singularity exec -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL224/anat_preproc_wf/surface_recon_wf/autorecon1

output 1

/mnt/Projects/CREST/sub-PIL224/anat_preproc_wf/surface_recon_wf/autorecon1 failed

command 2

singularity exec -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL224/anat_preproc_wf/anat_template_wf/t1_conform/mapflow/_t1_conform0/sub-PIL224_T1w_ras.nii.gz

output 2

/mnt/Projects/CREST/sub-PIL224/anat_preproc_wf/anat_template_wf/t1_conform/mapflow/_t1_conform0/sub-PIL224_T1w_ras.nii.gz failed

command 3

singularity exec -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL224/anat/sub-PIL224_desc-preproc_T1w.nii.gz

output 3

/mnt/Projects/CREST/sub-PIL224/anat/sub-PIL224_desc-preproc_T1w.nii.gz failed
jdkent commented 4 years ago

You were close, I was trying to suggest the original t1w file

Here's what I wanted:

singularity exec -B /Shared/vosslabhpc:/mnt /Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL224/anat/sub-PIL224_T1w.nii.gz

output:

/mnt/Projects/CREST/sub-PIL224/anat/sub-PIL224_T1w.nii.gz int16 [256, 240, 256] 1.00x1.40x1.00   sform

Could you do the same for a successful run?

ariveradompenciel commented 4 years ago

Oh sorry, thanks for clarifying! Here is the code and output for two successful runs:

singularity exec -B /Shared/vosslabhpc:/mnt /Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL220/anat/sub-PIL220_T1w.nii.gz /mnt/Projects/CREST/sub-PIL220/anat/sub-PIL220_T1w.nii.gz int16 [256, 240, 256] 1.00x1.00x1.00

singularity exec -B /Shared/vosslabhpc:/mnt /Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg nib-ls /mnt/Projects/CREST/sub-PIL227/anat/sub-PIL227_T1w.nii.gz

/mnt/Projects/CREST/sub-PIL227/anat/sub-PIL227_T1w.nii.gz int16 [256, 240, 256] 1.00x1.00x1.00

Seems sub-PIL224's anatomical image is of a different size…

jdkent commented 4 years ago

it's weird that sub-PIL224 voxel sizes are not isotropic (not the same number for all dimensions) 1.00x1.40x1.00, while the other participants are isotropic: 1.00x1.00x1.00.

I would look at the image in a viewer and see if the image looks stretched/compressed relative to other participants, or somehow check if the dimensions should be 1.00x.1.40x1.00 (do you have access to the original dicom images?).