search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
634 stars 293 forks source link

Talairach alignment error with new update? #656

Closed yidilozdemir closed 7 years ago

yidilozdemir commented 7 years ago

Hello all, For the fmriprep preprocessing I have been trying to run today on multiple participants, a Talairach alignment error shows up consistently, and makes the script exit. Just a couple of days ago I was able to run this docker on other subjects even though the procedure I used to get the nifti files for docker are the same. I thought this was a freesurfer-originated problem but I saw that yesterday you made a new update that concerns orientation of templates, so I am wondering if this error in any way could be coming from the new update. If so what I can do to override this other than completely overriding Talairach alignment? (there is a tag for that in freesurfer)

Here is some portion of that log if it helps: talairach done

 cp transforms/talairach.auto.xfm transforms/talairach.xfm 

#--------------------------------------------
#@# Talairach Failure Detection Wed Aug  9 17:31:33 UTC 2017
/out/freesurfer/sub-09/mri

 talairach_afd -T 0.005 -xfm transforms/talairach.xfm 

ERROR: talairach_afd: Talairach Transform: transforms/talairach.xfm ***FAILED*** (p=0.0000, pval=0.0000 < threshold=0.0050)

Manual Talairach alignment may be necessary, or
include the -notal-check flag to skip this test,
making sure the -notal-check flag follows -all
or -autorecon1 in the command string.
See:

http://surfer.nmr.mgh.harvard.edu/fswiki/FsTutorial/Talairach

ERROR: Talairach failed!

Linux 17a449efe2be 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

recon-all -s sub-09 exited with ERRORS at Wed Aug  9 17:31:33 UTC 2017

For more details, see the log file /out/freesurfer/sub-09/scripts/recon-all.log
To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting

Standard error:

Return code: 1
Interface ReconAll failed to run. 

170809-17:31:34,673 workflow INFO:
     ***********************************
170809-17:31:34,673 workflow ERROR:
     could not run node: fmriprep_wf.single_subject_09_wf.anat_preproc_wf.surface_recon_wf.autorecon1
170809-17:31:34,673 workflow INFO:
     crashfile: /out/fmriprep/sub-09/log/20170809-161613_2710c11b-6822-4dd8-87a4-09699f1b19bf/crash-20170809-173134-root-autorecon1-f6feac31-6ede-4b63-8a02-35208898c5a6.txt
170809-17:31:34,705 workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    load_entry_point('fmriprep==0.6.0', 'console_scripts', 'fmriprep')()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 161, in main
    create_workflow(opts)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 303, in create_workflow
    assert(report_errors > 0)
AssertionError
fmriprep: Please report errors to https://github.com/poldracklab/fmriprep/issues

Thank you so much! Idil

effigies commented 7 years ago

In general, overriding the talairach check will simply cause things to fail silently.

A few questions:

1) What was your command, and what does your data (BIDS) directory look like? 2) When you upgraded to a new version of fmriprep/fmriprep-docker, did you re-use existing work/output directories, or create new ones? 3) In the working directory, can you run mri_info (or fslhd) on fmriprep_wf/single_subject_09_wf/anat_preproc_wf/t1_conform/*.nii.gz? Screenshots of the image verifying that its orientation looks reasonable would be good, too. 4) Can you share the (partially completed) report sub-09.html?

Also, it appears you're using 0.6.0, not 1.0.0-rc1 (which is the same as 0.6.1... if you try to upgrade and get 0.6.1, everything is ok!).

yidilozdemir commented 7 years ago

I just realized I forgot to upgrade-sorry about that, I will do that and run it again as soon as possible.

  1. I used sudo fmriprep-docker --anat-only /home/lab/Desktop/freesurfer/ds001 /home/lab/Desktop/freesurfer/ds001 participant

  3. niiRead() gave an error opening file

  4. sub9.pdf

effigies commented 7 years ago

It's hard to tell with a PDF version of the report... I'm guessing you opened it in Firefox? If you can post the actual html file somewhere, I'll be able to see the images of the brain.

But at first glance, it looks like you have a T1w image with a bad affine. Can you view the original T1w image in fslview or freeview or mricron and post a screenshot?

yidilozdemir commented 7 years ago

Thank you for the help. Yes the orientation is crooked; I thought this would be something accounted for in the pipeline but I don't really know the steps. Is there a tag I could apply to the command line to correct this?

Also when I ran the same command line again, it continued and finished the process so the report right now looks complete, but when I actually viewed the hemispheres they were messed up in some ways; like the inflated surfaces would appear overlapped and back to back instead of two lobes. I made the upgrade and am trying it again.

t1-sub9 -this is the T1 image https://drive.google.com/open?id=0BwBa0ry6hC88djQzTktEQVpuekk -this is the public link to the html file

effigies commented 7 years ago

Unfortunately, we can't fix the orientation. We might be able to use some heuristic to make a guess, but it would be very difficult to verify that it wasn't left-right flipped. We really depend on the scanner properly encoding the affine matrix, and any pre-fmriprep preprocessing to preserve its integrity.

If you can find any earlier versions of your dataset where viewing the image shows that the axes are labelled correctly (that is, it's fine if it's rotated as you see, but the L/R, P/A, I/S marks really need to be consistent with the anatomy), then we can use that.

Finally, inflated surfaces are not tied to the anatomy. They're just for display. As long as your white and pial surfaces align, you're in reasonably good shape.

effigies commented 7 years ago

Closing this because it's not an FMRIPREP issue, but feel free to continue to follow-up on this thread, if there's something we can help you with.

yidilozdemir commented 7 years ago

Hello again,

There is a slight chance this new problem might be related to my previous preprocessing issues so this is why I am following up on this thread; in the subjects we corrected the T1 affine, we are constantly getting stuck at this step. It does not show any errors but when I look at the fmriprep log for the subject, I see a crash log with this Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 301, in _send_procs_to_workers
    jobid].hash_exists()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 244, in hash_exists
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 462, in _get_inputs
    output_value = results.outputs.get()[output_name]
AttributeError: 'NoneType' object has no attribute 'get'

Does this seem like an fmriprep related issue? This is the last job finished before script exits;

Executing node ds_mni_tpms in dir: /root/src/fmriprep/work/fmriprep_wf/single_subject_11_wf/anat_preproc_wf/anat_derivatives_wf/ds_mni_tpms
170811-20:47:11,6 workflow INFO:
     [Job finished] jobname: ds_mni_tpms jobid: 26
effigies commented 7 years ago

Hi. There's no way of saying what node failed from this. If you don't still have the log, you can pass --nthreads 1 to ensure that only one node runs at a time. It should be more obvious which is failing, then.

yidilozdemir commented 7 years ago

I will try that-thank you!

yidilozdemir commented 7 years ago

So it looks like subject summary node fails; fmriprep_wf.single_subject_12_wf.summary. I guess the problem depends on '_Undefined' object it mentions but I don't know what that refers to. Do you think this might be caused by something in my environment?

Thank you so much for all the help, Idil

170814-17:08:32,271 workflow INFO:
     Executing node summary in dir: /root/src/fmriprep/work/fmriprep_wf/single_subject_12_wf/summary
170814-17:08:32,994 workflow ERROR:
     ['Node summary failed to run on host 336103de74b8.']
170814-17:08:34,591 workflow INFO:
     Saving crash info to /out/fmriprep/sub-12/log/20170814-160455_51ddee26-26bf-4b72-a6a2-e1345afb08af/crash-20170814-170833-root-summary-42203582-2a33-4875-9438-24c8fd975c95.txt
170814-17:08:34,591 workflow INFO:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/linear.py", line 43, in run
    node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 372, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 613, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1083, in run
    runtime = self._run_wrapper(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1031, in _run_wrapper
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/reports.py", line 94, in _run_interface
    return super(SubjectSummary, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/reports.py", line 61, in _run_interface
    segment = self._generate_segment()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/reports.py", line 118, in _generate_segment
    for series in self.inputs.bold)
TypeError: '_Undefined' object is not iterable
Interface SubjectSummary failed to run. 

170814-17:08:34,604 workflow INFO:
     ***********************************
170814-17:08:34,604 workflow ERROR:
     could not run node: fmriprep_wf.single_subject_12_wf.summary
170814-17:08:34,604 workflow INFO:
     crashfile: /out/fmriprep/sub-12/log/20170814-160455_51ddee26-26bf-4b72-a6a2-e1345afb08af/crash-20170814-170833-root-summary-42203582-2a33-4875-9438-24c8fd975c95.txt
170814-17:08:34,604 workflow INFO:
     ***********************************
oesteban commented 7 years ago

Hi @yidilozdemir, could you please copy the contents of the crashfile?

out/fmriprep/sub-12/log/20170814-160455_51ddee26-26bf-4b72-a6a2-e1345afb08af/crash-20170814-170833-root-summary-42203582-2a33-4875-9438-24c8fd975c95.txt
yidilozdemir commented 7 years ago 
Node: fmriprep_wf.single_subject_12_wf.summary
Working directory: /root/src/fmriprep/work/fmriprep_wf/single_subject_12_wf/summary

Node inputs:

bold = <undefined>
ignore_exception = False
output_spaces = ['template', 'fsaverage5']
subject_id = sub-12
subjects_dir = /out/freesurfer
t1w = ['/data/sub-12/anat/sub-12_run-01_T1w.nii.gz']
t2w = <undefined>
template = MNI152NLin2009cAsym

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/linear.py", line 43, in run
    node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 372, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 613, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1083, in run
    runtime = self._run_wrapper(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1031, in _run_wrapper
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/reports.py", line 94, in _run_interface
    return super(SubjectSummary, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/reports.py", line 61, in _run_interface
    segment = self._generate_segment()
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/reports.py", line 118, in _generate_segment
    for series in self.inputs.bold)
TypeError: '_Undefined' object is not iterable
Interface SubjectSummary failed to run.
oesteban commented 7 years ago

Ok, that bold input should not be <undefined>. I think this looks like a bug.

Could you check at the beginning of the log? There should be a list of subjects and runs to be processed up there. Let's see what we got for that subject 12.

Also, are you reusing the work directory from a previous run?. If so, after checking the head of the log, please wipe the work directory out and re-run to check if there's a problem of caching here.

yidilozdemir commented 7 years ago

Hello, I ran quite a lot of things in between in the terminal and I cannot seem to go back that far. If it helps, this a different crash file from sub-12. Tomorrow I can run 12 with docker again to get that beginning. I use the same working directory but I discard the previous subject files after each analysis, although the fmriprep and freesurfer folders keep the unfinished subject files that I cannot alter. 

Node: fmriprep_wf.single_subject_12_wf.anat_preproc_wf.surface_recon_wf.autorecon1
Working directory: /root/src/fmriprep/work/fmriprep_wf/single_subject_12_wf/anat_preproc_wf/surface_recon_wf/autorecon1

Node inputs:

T1_files = ['/root/src/fmriprep/work/fmriprep_wf/single_subject_12_wf/anat_preproc_wf/t1_reorient/sub-12_run-01_T1w_ras_template_ras.nii.gz']
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon1
environ = {}
expert = <undefined>
flags = ['-noskullstrip']
hemi = <undefined>
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = False
ignore_exception = False
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 7
parallel = <undefined>
subject_id = recon_all
subjects_dir = /out/freesurfer
talairach = <undefined>
terminal_output = stream
use_T2 = <undefined>
xopts = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 301, in _send_procs_to_workers
    jobid].hash_exists()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 244, in hash_exists
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 462, in _get_inputs
    output_value = results.outputs.get()[output_name]
AttributeError: 'NoneType' object has no attribute 'get'
effigies commented 7 years ago

It's possible for SubjectSummary().inputs.bold == Undefined if the BIDS data grabber can't find any BOLD files. Can you run tree or similar on your sub-12 data directory and output that?

yidilozdemir commented 7 years ago

I am running --anat-only and I don't have any BOLD files. 

lab@lab-ThinkStation-E31:~/Desktop/freesurfer/ds001/sub-12$ tree
.
├── anat
│   └── sub-12_run-01_T1w.nii.gz
└── func

lab@lab-ThinkStation-E31:~/Desktop/freesurfer/ds001/fmriprep/sub-12$ tree
.
├── anat
│   ├── sub-12_run-01_T1w_brainmask.nii.gz
│   ├── sub-12_run-01_T1w_class-CSF_probtissue.nii.gz
│   ├── sub-12_run-01_T1w_class-GM_probtissue.nii.gz
│   ├── sub-12_run-01_T1w_class-WM_probtissue.nii.gz
│   ├── sub-12_run-01_T1w_dtissue.nii.gz
│   ├── sub-12_run-01_T1w_preproc.nii.gz
│   ├── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_brainmask.nii.gz
│   ├── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_class-CSF_probtissue.nii.gz
│   ├── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_class-GM_probtissue.nii.gz
│   ├── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_class-WM_probtissue.nii.gz
│   ├── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_dtissue.nii.gz
│   ├── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_preproc.nii.gz
│   └── sub-12_run-01_T1w_space-MNI152NLin2009cAsym_warp.h5
└── log
    ├── 20170814-143843_3c37f2ee-39e8-4439-826f-148f3027f79a
    │   ├── crash-20170814-143849-root-ds_summary_report-32eefcec-d917-431f-ba55-35267495f6b4.txt
    │   ├── crash-20170814-143849-root-ds_summary_report-4700571f-f1e8-4de8-9c73-ce221a725789.txt
    │   ├── crash-20170814-143849-root-summary-c24f9720-97d6-4ded-b389-184fb79c1e56.txt
    │   ├── crash-20170814-154539-root-autorecon1-7d6b9654-a52a-453a-8085-6d186d75e609.txt
    │   └── crash-20170814-154540-root-autorecon1-60b991fc-4ed4-4be2-8a22-2d5a02966103.txt
    └── 20170814-160455_51ddee26-26bf-4b72-a6a2-e1345afb08af
        └── crash-20170814-170833-root-summary-42203582-2a33-4875-9438-24c8fd975c95.txt

I don't have a sub-12 folder formed inside the freesurfer folder.

oesteban commented 7 years ago

The func folder seems to be empty, is that correct?

yidilozdemir commented 7 years ago

Yes, we don't need functional data for the procedure.

effigies commented 7 years ago

Okay. I need to account for no functional data. Sorry about that.

effigies commented 7 years ago

Actually, @yidilozdemir, can you verify that using the --anat-only flag doesn't resolve the issue?

yidilozdemir commented 7 years ago

I have always used that tag. Glad I could help!

effigies commented 7 years ago

Moved this conversation to #669.