search

nipreps / mriqc

Automated Quality Control and visual reports for Quality Assessment of structural (T1w, T2w) and functional MRI of the brain
http://mriqc.readthedocs.io
Apache License 2.0
297 stars 132 forks source link

running mriqc in singularity: Interface RobustMNINormalizationRPT failed to run #433

Closed zhenghanQ closed 7 years ago

zhenghanQ commented 7 years ago

I tried the singularity image of mriqc on our HPC to process my imaging data for the first time. I saw the following error a number of times:

Traceback (most recent call last): File "/root/src/mriqc/src/nipype/nipype/pipeline/plugins/multiproc.py", line 52, in run_node result['result'] = node.run(updatehash=updatehash) File "/root/src/mriqc/src/nipype/nipype/pipeline/engine/nodes.py", line 366, in run self._run_interface() File "/root/src/mriqc/src/nipype/nipype/pipeline/engine/nodes.py", line 476, in _run_interface self._result = self._run_command(execute) File "/root/src/mriqc/src/nipype/nipype/pipeline/engine/nodes.py", line 607, in _run_command result = self._interface.run() File "/root/src/mriqc/src/nipype/nipype/interfaces/base.py", line 1085, in run runtime = self._run_wrapper(runtime) File "/root/src/mriqc/src/nipype/nipype/interfaces/base.py", line 1033, in _run_wrapper runtime = self._run_interface(runtime) File "/root/src/mriqc/src/niworkflows/niworkflows/common/report.py", line 72, in _run_interface self._generate_report() File "/root/src/mriqc/src/niworkflows/niworkflows/common/report.py", line 175, in _generate_report compress=self.inputs.compress_report), File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 318, in plot_registration svg = extract_svg(display, compress=compress) File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 146, in extract_svg image_svg = svg_compress(image_svg, compress) File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 77, in svg_compress if (_which('svgo') and compress == 'auto') or compress == True: File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 406, in _which subprocess.run([cmd], stdin=subprocess.DEVNULL, AttributeError: 'module' object has no attribute 'run' Interface RobustMNINormalizationRPT failed to run.

chrisgorgo commented 7 years ago

Interesting - it seems that for some reason wrong version of Python (presumably the one installed on the host, instead of the one inside the container) was used.

  1. Do you have PYTHONPATH set?
  2. How did you create the singularity container (could you paste the command line)?
  3. Which version of mriqc is this?
zhenghanQ commented 7 years ago
  1. this is the PYTHONPATH in the container: /usr/local/miniconda/lib/python2.7/site-packages
  2. I used the following command (according to the instruction on fmriprep): 
    $ docker run --privileged -t --rm \
-v /var/run/docker.sock:/var/run/docker.sock \
-v D:\host\path\where\to\output\singularity\image:/output \
singularityware/docker2singularity \
poldracklab/mriqc:latest
  3. I am using this version: poldracklab_mriqc_latest-2017-03-14-e17fb1c459b8.img
chrisgorgo commented 7 years ago

I see. One of our containers might be defective. Could you try

$ docker run --privileged -t --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v D:\host\path\where\to\output\singularity\image:/output \
    singularityware/docker2singularity \
    poldracklab/mriqc:0.9.0-1-python35
chrisgorgo commented 7 years ago

We've just released a new version so you can try:

$ docker run --privileged -t --rm \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v D:\host\path\where\to\output\singularity\image:/output \
    singularityware/docker2singularity \
    poldracklab/mriqc:0.9.1
zhenghanQ commented 7 years ago

I tried the version 0.9.0 and saw a few crash reports with the following error message. I'll try the newest released version later today. Thanks!

Traceback (most recent call last):
  File "/om/user/zqi/projects/CASL/Analysis/env/bin/nipype_display_crash", line 85, in <module>
    debug, args.directory)
  File "/om/user/zqi/projects/CASL/Analysis/env/bin/nipype_display_crash", line 17, in display_crash_files
    crash_data = loadcrash(crashfile)
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/nipype/utils/filemanip.py", line 489, in loadcrash
    return loadpkl(infile)
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/nipype/utils/filemanip.py", line 514, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 1384, in load
    return Unpickler(file).load()
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 1223, in load_build
    setstate(state)
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/traits/has_traits.py", line 1441, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/traits/has_traits.py", line 1544, in trait_set
    setattr( self, name, value )
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/nipype/interfaces/base.py", line 2047, in validate
    value = super(MultiPath, self).validate(object, name, newvalue)
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/traits/trait_types.py", line 2337, in validate
    return TraitListObject( self, object, name, value )
  File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site-packages/traits/trait_handlers.py", line 2313, in __init__
    raise excp
traits.trait_errors.TraitError: Each element of the 'moving_image' trait of a RobustMNINormalizationInputSpecRPT instance must be an existing file name, but a value of u'/mnt/scratch/Mon/zqi/workflow_enumerator/anatMRIQCT1w/AFNISkullStripWorkflow/_in_file_..mnt..user..zqi..projects..CASL..Results..Imaging..openfmri..sub-CASL13100..ses-post..anat..sub-CASL13100_ses-post_run-001_T1w.nii.gz/CorrectINU/sub-CASL13100_ses-post_run-001_T1w_conformed_corrected.nii.gz' <type 'unicode'> was specified.
chrisgorgo commented 7 years ago

This is very bizarre. What is $PATH set to inside and outside the container?

On Mar 22, 2017 6:49 AM, "zhenghanQ" notifications@github.com wrote:

I tried the version 0.9.0 and saw a few crash reports with the following error message. I'll try the newest released version later today. Thanks! Traceback (most recent call last): File "/om/user/zqi/projects/CASL/Analysis/env/bin/nipype_display_crash", line 85, in debug, args.directory) File "/om/user/zqi/projects/CASL/Analysis/env/bin/nipype_display_crash", line 17, in display_crash_files crash_data = loadcrash(crashfile) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/nipype/utils/filemanip.py", line 489, in loadcrash return loadpkl(infile) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/nipype/utils/filemanip.py", line 514, in loadpkl unpkl = pickle.load(pkl_file) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 1384, in load return Unpickler(file).load() File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 864, in load dispatchkey http://self File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 1223, in load_build setstate(state) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/traits/has_traits.py", line 1441, in setstate self.trait_set( trait_change_notify = trait_change_notify, *state ) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/traits/has_traits.py", line 1544, in trait_set setattr( self, name, value ) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/nipype/interfaces/base.py", line 2047, in validate value = super(MultiPath, self).validate(object, name, newvalue) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/traits/trait_types.py", line 2337, in validate return TraitListObject( self, object, name, value ) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/ site-packages/traits/trait_handlers.py", line 2313, in init raise excp traits.trait_errors.TraitError: Each element of the 'moving_image' trait of a RobustMNINormalizationInputSpecRPT instance must be an existing file name, but a value of u'/mnt/scratch/Mon/zqi/workflow_enumerator/ anatMRIQCT1w/AFNISkullStripWorkflow/in_file*..mnt..user..zqi..projects.. CASL..Results..Imaging..openfmri..sub-CASL13100..ses- post..anat..sub-CASL13100_ses-post_run-001_T1w.nii.gz/ CorrectINU/sub-CASL13100_ses-post_run-001_T1w_conformed_corrected.nii.gz' <type 'unicode'> was specified.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/poldracklab/mriqc/issues/433#issuecomment-288403617, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOkp0d4kfjLlOttnjlfr8j6U7SstUb9ks5roSb9gaJpZM4MkbCN .

chrisgorgo commented 7 years ago

Do you have PYTHONPATH set outside the container? Can you try unsetting it before running?

On Mar 22, 2017 7:06 AM, "Chris Gorgolewski" < krzysztof.gorgolewski@gmail.com> wrote:

This is very bizarre. What is $PATH set to inside and outside the container?

On Mar 22, 2017 6:49 AM, "zhenghanQ" notifications@github.com wrote:

I tried the version 0.9.0 and saw a few crash reports with the following error message. I'll try the newest released version later today. Thanks! Traceback (most recent call last): File "/om/user/zqi/projects/CASL/Analysis/env/bin/nipype_display_crash", line 85, in debug, args.directory) File "/om/user/zqi/projects/CASL/Analysis/env/bin/nipype_display_crash", line 17, in display_crash_files crash_data = loadcrash(crashfile) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/nipype/utils/filemanip.py", line 489, in loadcrash return loadpkl(infile) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/nipype/utils/filemanip.py", line 514, in loadpkl unpkl = pickle.load(pkl_file) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 1384, in load return Unpickler(file).load() File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 864, in load dispatchkey http://self File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/pickle.py", line 1223, in load_build setstate(state) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/traits/has_traits.py", line 1441, in setstate self.trait_set( trait_change_notify = trait_change_notify, *state ) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/traits/has_traits.py", line 1544, in trait_set setattr( self, name, value ) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/nipype/interfaces/base.py", line 2047, in validate value = super(MultiPath, self).validate(object, name, newvalue) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/traits/trait_types.py", line 2337, in validate return TraitListObject( self, object, name, value ) File "/om/user/zqi/projects/CASL/Analysis/env/lib/python2.7/site- packages/traits/trait_handlers.py", line 2313, in init raise excp traits.trait_errors.TraitError: Each element of the 'moving_image' trait of a RobustMNINormalizationInputSpecRPT instance must be an existing file name, but a value of u'/mnt/scratch/Mon/zqi/workflo w_enumerator/anatMRIQCT1w/AFNISkullStripWorkflow/in_file*.. mnt..user..zqi..projects..CASL..Results..Imaging..openfmri.. sub-CASL13100..ses-post..anat..sub-CASL13100_ses-post_run- 001_T1w.nii.gz/CorrectINU/sub-CASL13100_ses-postrun-001 T1w_conformed_corrected.nii.gz' <type 'unicode'> was specified.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/poldracklab/mriqc/issues/433#issuecomment-288403617, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOkp0d4kfjLlOttnjlfr8j6U7SstUb9ks5roSb9gaJpZM4MkbCN .

zhenghanQ commented 7 years ago

I did see all the resulting report html files though. mriqc reports look really nice!!

I used this command to run mriqc

singularity exec -B /om:/mnt -c poldracklab_mriqc_0.9.0-1-python35-2017-03-14-19a236acac82.img mriqc --participant_label sub# -w /mnt/workdir /mnt/bidsdatadir/ /mnt/qcoutdir/ participant

Inside the container: echo $PATH /usr/local/miniconda/bin:/opt/c3d:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/ants:/usr/lib/fsl/5.0:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin echo $PYTHONPATH /usr/local/miniconda/lib/python3.5/site-packages

Why would outside PYTHONPATH affect the environment inside the container?

chrisgorgo commented 7 years ago

I'm glad you like the reports!

I think I know what is going on. The error you are seeing is not the actual crash, but the inability of load the crashfile (due to python version difference inside and outside the container - see https://github.com/nipy/nipype/issues/1184). We will switch to using plain text crash files in the next release (https://github.com/nipy/nipype/pull/1885), but in the meantime could you a) open the crashfile using nipype_display_crash in the container OR b) copy the relecant part of the console output with the corresponding error?

What percentage of the subjects did it fail on?

zhenghanQ commented 7 years ago

The error message I pasted in the previous response was from the crash file. The crash occurred in EPI2MNI and SpatialNormalization.

I have only run one participant so far.

oesteban commented 7 years ago

And yet, are you able to see the final reports for that participant?

zhenghanQ commented 7 years ago

yes, I am able to see these final reports for that participant: T1, resting state and task fMRI, but not dwi.

chrisgorgo commented 7 years ago

To show the source of the error you need to use nipype from the container to display the crashfile instead of the one on your system

singularity exec -B /om:/mnt -c poldracklab_mriqc_0.9.0-1-python35-2017-03-14-19a236acac82.img nipypecli crash /mnt/path/to/your/crashfile.pklz

I'm also curious how you are getting the reports despite the crashes...

oesteban commented 7 years ago

Thanks:

Could you verify if you are getting these and interpreting them as ERRORs? (otherwise it is puzzling that you are getting final reports...)

Thanks a lot.

chrisgorgo commented 7 years ago

MRIQC does not support DWI (yet!). Are there any runs missing?

zhenghanQ commented 7 years ago

I used Mosh to run mriqc. So I lost most of the console outputs. I was only able to keep track of the crash files via the timestamp of the report. One thing I have to admit is that in order to save the runtime, I reused the working dir from the first run when I used the latest mriqc release. I don't know whether that will cause any issue. I plan rerunning mriqc with the new version and a new working dir.

okay, here is one of the crash files.:

File: /mnt/user/zqi/projects/CASL/Results/Imaging/qc/logs/crash-20170322-020933-zqi-EPI2MNI.a05-0d6e9092-a6ae-48de-b025-6190d421b5ca.pklz
Node: workflow_enumerator.funcMRIQC.SpatialNormalization.EPI2MNI.a05
Working directory: /mnt/scratch/Mon/zqi/workflow_enumerator/funcMRIQC/SpatialNormalization/_in_file_..mnt..user..zqi..projects..CASL..Results..Imaging..openfmri..sub-CASL13100..ses-pre..func..sub-CASL13100_ses-pre_task-mb_run-001_bold.nii.gz/EPI2MNI

Node inputs:

compress_report = auto
explicit_masking = True
generate_report = True
ignore_exception = False
moving = EPI
moving_image = ['/mnt/scratch/Mon/zqi/workflow_enumerator/funcMRIQC/SpatialNormalization/_in_file_..mnt..user..zqi..projects..CASL..Results..Imaging..openfmri..sub-CASL13100..ses-pre..func..sub-CASL13100_ses-pre_task-mb_run-001_bold.nii.gz/EPIApplyMask/sub-CASL13100_ses-pre_task-mb_run-001_bold_volreg_tstat_corrected_masked.nii.gz']
moving_mask = <undefined>
num_threads = 6
orientation = RAS
out_report = report.svg
reference = T1
reference_image = ['/root/src/mriqc/mriqc/data/mni/2mm_T2_brain.nii.gz']
reference_mask = <undefined>
settings = <undefined>
template = mni_icbm152_nlin_asym_09c
template_resolution = 1
testing = False

Traceback: 
Traceback (most recent call last):
  File "/root/src/mriqc/src/nipype/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/root/src/mriqc/src/nipype/nipype/pipeline/engine/nodes.py", line 366, in run
    self._run_interface()
  File "/root/src/mriqc/src/nipype/nipype/pipeline/engine/nodes.py", line 476, in _run_interface
    self._result = self._run_command(execute)
  File "/root/src/mriqc/src/nipype/nipype/pipeline/engine/nodes.py", line 607, in _run_command
    result = self._interface.run()
  File "/root/src/mriqc/src/nipype/nipype/interfaces/base.py", line 1085, in run
    runtime = self._run_wrapper(runtime)
  File "/root/src/mriqc/src/nipype/nipype/interfaces/base.py", line 1033, in _run_wrapper
    runtime = self._run_interface(runtime)
  File "/root/src/mriqc/src/niworkflows/niworkflows/common/report.py", line 72, in _run_interface
    self._generate_report()
  File "/root/src/mriqc/src/niworkflows/niworkflows/common/report.py", line 175, in _generate_report
    compress=self.inputs.compress_report),
  File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 318, in plot_registration
    svg = extract_svg(display, compress=compress)
  File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 146, in extract_svg
    image_svg = svg_compress(image_svg, compress)
  File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 77, in svg_compress
    if (_which('svgo') and compress == 'auto') or compress == True:
  File "/root/src/mriqc/src/niworkflows/niworkflows/viz/utils.py", line 406, in _which
    subprocess.run([cmd], stdin=subprocess.DEVNULL,
AttributeError: 'module' object has no attribute 'run'
Interface RobustMNINormalizationRPT failed to run.
zhenghanQ commented 7 years ago

all the task fmri reports are there. But they are not split into separate runs. the filename looks like sub-#_ses-#_task-taskname_bold.html. Does mriqc combine task runs into one report file?

chrisgorgo commented 7 years ago

This seems like an error generated by the old container.

Please try running with fresh working and output directories using the latest container (0.9.1) and see if everything goes well.

To answer your question - multiple runs are not combined in MRIQC. Each _bold.nii.gz file should result in a separate _bold.html file. For example sub-ds205s07_task-view_run-01_bold.html.

zhenghanQ commented 7 years ago

okay, I'll report back.

chrisgorgo commented 7 years ago

Thanks! Your feedback is much appreciated.

zhenghanQ commented 7 years ago

The latest version of mriqc completed without any crash files! However, I still only see one report per task, rather than per run per task. Same with .json files in derivitives.

chrisgorgo commented 7 years ago

This is strange. We constantly run tests with multiple runs and have not seen anything like this.

  1. Could you share the full console output (stdout)?
  2. Could you show what file/folder structure does your a) dataset have b) the mriqc outputs have. The 'tree' command is very good for this.

Best, Chris

On Wed, Mar 22, 2017 at 9:09 PM, zhenghanQ notifications@github.com wrote:

The latest version of mriqc completed without any crash files! However, I still only see one report per task, rather than per run per task. Same with .json files in derivitives.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/poldracklab/mriqc/issues/433#issuecomment-288612200, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOkp1gxG2zmG5QxuBeCW4c6v2Elc0wtks5rofBggaJpZM4MkbCN .

zhenghanQ commented 7 years ago

Sorry, I was traveling in conferences earlier. I am attaching the full console output (casl13102.txt), the file/folder structure for my bids dataset (data_tree.txt) and the file/folder structure of the mriqc output (qc_tree.txt). casl13102.txt data_tree.txt qc_tree.txt

oesteban commented 7 years ago

Hi @zhenghanQ,

Looking at your inputs/outputs I have the following comments:

Are you still experiencing problems?

zhenghanQ commented 7 years ago

@oesteban thanks very much! Now I noticed those missing hyphens. Interestingly, bids-validator did not report those issues. But I suspect this explain why mriqc only generates one report per task. However, fmriprep seems to handle these files fine. I got all the derivatives for each run.

chrisgorgo commented 7 years ago

@zhenghanQ Are you sure bids-validator did not give you any warnings about files not elonging to BIDS? For example for sub-CASL13106_ses-post_task-sent_run001_bold.nii.gz (which should've been sub-CASL13106_ses-post_task-sent_run-001_bold.nii.gz).

zhenghanQ commented 7 years ago

I ran bids-validator many days ago and might have only looked for error messages but not warning messages. I just reran bids-validator and noticed that these files with missing hyphens were indeed mentioned in the warning messages. I also have one error message telling me I need to name the task for my resting-state scans (see below)... But this might need to go to the bids-validator issue?

/sub-CASL13100/ses-post/func/sub-CASL13100_ses-post_task-rest_run-001_bold.nii.gz
            You have to define 'TaskName' for this file. It can be included one of the following locations: /task-rest_bold.json, /sub-CASL13100/sub-CASL13100_task-rest_bold.json, /sub-CASL13100/ses-post/sub-CASL13100_ses-post_task-rest_bold.json, /sub-CASL13100/ses-post/func/sub-CASL13100_ses-post_task-rest_run-001_bold.json
chrisgorgo commented 7 years ago

Thanks for checking.

In BIDS "resting state" is considered a cognitive paradigm like every other. Therefore you still need to define the "TaskName" field in the corresponding JSON file - (even if the value would be as trivial as "rest").

On Tue, Apr 4, 2017 at 7:27 PM, zhenghanQ notifications@github.com wrote:

I ran bids-validator many days ago and might have only looked for error messages but not warning messages. I just reran bids-validator and noticed that these files with missing hyphens were indeed mentioned in the warning messages. I also have one error message telling me I need to name the task for my resting-state scans (see below)... But this might need to go to the bids-validator issue?

      You have to define 'TaskName' for this file. It can be included one of the following locations: /task-rest_bold.json, /sub-CASL13100/sub-CASL13100_task-rest_bold.json, /sub-CASL13100/ses-post/sub-CASL13100_ses-post_task-rest_bold.json, /sub-CASL13100/ses-post/func/sub-CASL13100_ses-post_task-rest_run-001_bold.json```

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/poldracklab/mriqc/issues/433#issuecomment-291722086, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOkp7MYIf6rl5Fv-WfnTT7Nn5HaCQljks5rsvwlgaJpZM4MkbCN .

zhenghanQ commented 7 years ago

Thanks for all your help @chrisfilo @oesteban ! I think this issue can be closed now!

oesteban commented 7 years ago

Thank you for using MRIQC, don't hesitate to reopen if this happens again.