search

HBClab / SoftwareIssues

Issues when running primarily fmriprep/mriqc/xcpengine (primarily on argon)
0 stars 0 forks source link

CREST fmriprep: ICA-AROMA failed (FAST1050, job 1320777) #17

Open ariveradompenciel opened 4 years ago

ariveradompenciel commented 4 years ago

User = sodoma

Script/Job File

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M adriana-rivera-dompenciel@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg \
/mnt/Projects/CREST/ /mnt/Projects/CREST/derivatives \
participant --participant_label FAST1050 \
-w /nfsscratch/Users/sodoma/work/CRESTfmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

Email Message

Job 1320777 (sub-FAST1050.job) Complete
User             = sodoma
Queue            = UI@argon-itf-cf45-34.hpc
Host             = argon-itf-cf45-34.hpc
Start Time       = 06/03/2020 01:12:22
End Time         = 06/03/2020 09:52:22
User Time        = 00:24:29
System Time      = 00:00:08
Wallclock Time   = 08:40:00
CPU              = 21:52:20
Max vmem         = 8.999G
Exit Status      = 1

Output File

200603-04:55:18,536 nipype.interface WARNING:
     q-forms of reference and mask are substantially different

200603-09:51:55,484 nipype.interface WARNING:
     No signal components were classified
200603-09:51:55,489 nipype.workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_FAST1050_wf.func_preproc_task_rest_wf.ica_aroma_wf.ica_aroma_confound_extraction" (/nfsscratch/Users/sodoma/work/CRESTfmriprep/fmriprep_wf/single_subject_FAST1050_wf/func_preproc_task_rest_wf/ica_aroma_wf/ica_aroma_confound_extraction)
200603-09:51:57,53 nipype.workflow ERROR:
     Node ica_aroma_confound_extraction failed to run on host argon-itf-cf45-34.hpc.
200603-09:51:57,58 nipype.workflow ERROR:
     Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-FAST1050/log/20200603-011231_51b53cc0-ffe7-4477-83f8-dd3784fbdb2c/crash-20200603-095157-sodoma-ica_aroma_confound_extraction-71b82bc1-7eb8-46c8-953d-7ab2739ee2f3.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/confounds.py", line 123, in _run_interface
    raise RuntimeError('ICA-AROMA failed')
RuntimeError: ICA-AROMA failed

200603-09:52:14,300 nipype.workflow INFO:
     [Node] Finished "fmriprep_wf.single_subject_FAST1050_wf.func_preproc_task_rest_wf.ica_aroma_wf.add_nonsteady".
200603-09:52:20,981 nipype.workflow ERROR:
     could not run node: fmriprep_wf.single_subject_FAST1050_wf.func_preproc_task_rest_wf.ica_aroma_wf.ica_aroma_confound_extraction

Error (stderr file)

/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
/usr/local/miniconda/lib/python3.6/site-packages/bids/grabbids/__init__.py:6: FutureWarning: grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8
  warnings.warn("grabbids has been renamed to layout in version 0.6.5, and will be removed in version 0.8", FutureWarning)
/usr/local/miniconda/lib/python3.6/site-packages/bids/layout/bids_layout.py:116: UserWarning: 'dataset_description.json' file is missing from project root. You may want to set the root path to a valid BIDS project.
  warnings.warn("'dataset_description.json' file is missing from "
/usr/local/miniconda/lib/python3.6/site-packages/nitime/utils.py:980: FutureWarning: Conversion of the second argument of issubdtype from `complex` to `np.complexfloating` is deprecated. In future, it will be treated as `np.complex128 == np.dtype(complex).type`.
  complex_result = (np.issubdtype(in1.dtype, np.complex) or
/usr/local/miniconda/lib/python3.6/site-packages/nitime/utils.py:981: FutureWarning: Conversion of the second argument of issubdtype from `complex` to `np.complexfloating` is deprecated. In future, it will be treated as `np.complex128 == np.dtype(complex).type`.
  np.issubdtype(in2.dtype, np.complex))
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/matplotlib/contour.py:1173: UserWarning: No contour levels were found within the data range.
  warnings.warn("No contour levels were found"
/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py:1022: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  betas = np.linalg.lstsq(X, data.T)[0]
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nilearn/image/resampling.py:518: UserWarning: Casting data from int32 to float32
  warnings.warn("Casting data from %s to %s" % (data.dtype.name, aux))
/usr/local/miniconda/lib/python3.6/site-packages/nipype/algorithms/confounds.py:1022: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  betas = np.linalg.lstsq(X, data.T)[0]
Errors occurred while generating reports for participants: FAST1050 (1).
jdkent commented 4 years ago

This is always a fun error. The most relevant bit is: No signal components were classified from ICA-AROMA, meaning according to ICA-AROMA, all the components derived were classified as noise.

In the past, this has meant the brainmask was unexpectedly misshapen or the transformation from bold space (native) to template space (MNI) was severely sub-optimal.

I haven't seen this before: q-forms of reference and mask are substantially different

which could be a hint on what to look for, and that message is in #9 as well.

It will be helpful to have access to this participant's working directory and look at some of the intermediate outputs (I think this directory): /nfsscratch/Users/sodoma/work/CRESTfmriprep/fmriprep_wf/single_subject_FAST1050_wf/

Could this directory be copied somewhere on the server that we all can access?