mindboggle table output issues

[satra]

1. ants thickinthehead has no output (just the header row).
2. frontomarginal sulcus is 0 for everybody
3. the csv files are all strings (i.e. requires me to look and figure out what's a number, a label, and a list). if you are going to store numbers and lists, this should be json
4. some of the csv files are not formatted consistently or properly, they have spaces with numbers

[binarybottle]

Thank you, Satra!

First let's tackle the thickinthehead:

I just ran thickinthehead from within iPython and it worked just fine, outputting to the command line and to a table reasonable values. i also ran the following command:

mindboggle Twins-2-2 --ants /Users/arno/antsCorticalThickness/Twins-2-2/antsBrainSegmentation.nii.gz --thickness --no_surfaces -p 3

This command also completed without error, and the resulting ants thickinthehead table looks fine. When looking at the github repo, I see that nothing's changed in io_tables.py or ants.py (thickinthehead) or LABELS.py or LUT.py since November, and you're running a version of mindboggle post jan 5th, so I don't think it is due to any changes in the main mindboggle program either. I assume you ran the --all option, in which case I will run that next...

[binarybottle]

Just to be sure, have you tried running the most recent version of mindboggle?

[satra]

i'm using this version:

7cecab8d1ea5c9ed2975728a67e38320be166fa0 Date: Sun Feb 8 21:29:43 2015 -0800

also it looks like some of the volumes do not contain the full assortment of labels.

img = nb.load('/om/scratch/Mon/satra/SAD/MBwork/SAD_020/Mindboggle/Volume_labels/FreeSurfer_cerebellum_ANTs_cerebrum/wmparc.mgz.nii.gz_to_ANTs_labels.nii.gz_to_ANTs_labels.nii.gz_through_combined_segmentations.nii.gz') 

In [5]: np.unique(img.get_data())
Out[5]: 
array([    0.,     7.,     8.,    46.,    47.,  2012.,  2019.,  2020.,
        2027.,  2028.], dtype=float32)

[binarybottle]

I believe that was an ephemeral bug that you happened to catch in your net! You need a more recent version to squash it!!!

[satra]

current master fails to run:

TypeError: relabel_surface() got an unexpected keyword argument 'hemi'
Interface Function failed to run. 

[binarybottle]

I am sorry, Satra -- it should work now.

[satra]

Exception: Some connections were not found 
Module Surface_labels has no input called Reindex_labels.hemi

[binarybottle]

Yeesh -- I removed this second reference to hemi as well.

Please try again!

[satra]

more errors!

    >>> relabel_surface(vtk_file, old_labels, new_labels, erase_remaining, erase_labels, erase_value, output_file)
    >>> # View
    >>> plot_surfaces('relabeled_rh.labels.DKT31.manual.vtk')

    """  
    import os
    import numpy as np  
    from mindboggle.utils.io_vtk import read_vtk, write_vtk  

    # Load labeled vtk surfaces:
    faces, lines, indices, points, npoints, scalars, \
        name, input_vtk = read_vtk(vtk_file, return_first=True, return_array=True)
    new_scalars = scalars[:]

    # Raise an error if inputs set incorrectly:
    if (new_labels and not old_labels) or \
            (len(old_labels) != len(new_labels)) or \
            (erase_remaining and not old_labels):
        raise IOError("Please check inputs for relabel_surface().")

the error above is raised with these inputs below

> args
vtk_file = /om/scratch/Mon/satra/mdd/MBwork/50250/Mindboggle/Surface_labels/_hemi_lh/FreeSurfer_annot_to_vtk/lh.aparc.vtk
old_labels = [1002, 1003, 1005, 1006, 1007, 1008, 1009, 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, 1018, 1019, 1020, 1021, 1022, 1023, 1024, 1025, 1026, 1027, 1028, 1029, 1030, 1031, 1034, 1035, 2002, 2003, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026, 2027, 2028, 2029, 2030, 2031, 2034, 2035]
new_labels = []
erase_remaining = True
erase_labels = [0]
erase_value = -1
output_file = 

[binarybottle]

Great catch, Satra -- thank you! I have been using this function with an equal number of old and new labels, so I didn't catch the silly error in the conditional statement that checks if they're the same size -- this of course doesn't make sense if you don't specify new labels. Anyway, I am sorry to ask you to try again, but please try again!

[satra]

Issue 1

this output is not getting any input: 'features.@fundus_per_sulcus': None,

Issue 2

and this file is still empty:

'tables.@thicknesses_of_ANTs_filled_cortex_labels': '/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_feature_shapes/ANTs_filled_cortex_label_thicknesses/thickinthehead_ANTs_labels.csv'

inputs to the routines that are generating this file is only seeing:

np.unique(nb.load('/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_labels/FreeSurfer_cerebellum_ANTs_cerebrum/wmparc.mgz.nii.gz_to_ANTs_labels.nii.gz_to_ANTs_labels.nii.gz_through_combined_segmentations.nii.gz').get_data())
Out[3]: array([  0.,   7.,   8.,  46.,  47.], dtype=float32)

np.unique(nb.load('/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_labels/Extract_FreeSurfer_cerebella/wmparc.mgz.nii.gz').get_data())
Out[6]: array([ 0,  7,  8, 46, 47], dtype=int32)

and this is the empty chain

np.unique(nb.load('/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_labels/Combine_ANTs_cortex_noncortex_labels/ANTs_labels.nii.gz_to_ANTs_labels.nii.gz_through_combined_segmentations.nii.gz').get_data())
Out[15]: array([ 0.], dtype=float32)

np.unique(nb.load('/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_labels/Fill_cortex_with_ANTs_labels/ANTs_labels.nii.gz_through_combined_segmentations.nii.gz').get_data())
Out[18]: array([ 0.], dtype=float32)

np.unique(nb.load('/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_labels/Extract_ANTs_cortex_labels/ANTs_labels.nii.gz').get_data())
Out[21]: array([ 0.])
np.unique(nb.load('/om/scratch/Mon/satra/mdd/MBwork/50207/Mindboggle/Volume_labels/antsApplyTransforms/ANTs_labels.nii.gz').get_data())
Out[23]: array([ 0.])

np.unique(nb.load('/om/user/satra/projects/mdd/set1/MBcache/ef2b6f7116e05b46746c044bfa995546/OASIS-TRT-20_jointfusion_DKT31_CMA_labels_in_MNI152_v2.nii.gz').get_data())
Out[25]: 
array([    0.,     4.,     5.,     6.,     7.,    10.,    11.,    12.,
          13.,    14.,    15.,    16.,    17.,    18.,    24.,    26.,
          28.,    30.,    43.,    44.,    45.,    46.,    49.,    50.,
          51.,    52.,    53.,    54.,    58.,    60.,    91.,    92.,
         630.,   631.,   632.,  1002.,  1003.,  1005.,  1006.,  1007.,
        1008.,  1009.,  1010.,  1011.,  1012.,  1013.,  1014.,  1015.,
        1016.,  1017.,  1018.,  1019.,  1020.,  1021.,  1022.,  1023.,
        1024.,  1025.,  1026.,  1027.,  1028.,  1029.,  1030.,  1031.,
        1034.,  1035.,  2002.,  2003.,  2005.,  2006.,  2007.,  2008.,
        2009.,  2010.,  2011.,  2012.,  2013.,  2014.,  2015.,  2016.,
        2017.,  2018.,  2019.,  2020.,  2021.,  2022.,  2023.,  2024.,
        2025.,  2026.,  2027.,  2028.,  2029.,  2030.,  2031.,  2034.,
        2035.], dtype=float32)

so the antsApplyTransforms step is taking the right set of inputs and then generating empty labels.

here are the transforms i see, and that doesn't make sense:

stdout 2015-03-07T01:29:21.599079:  1. inverse of /om/user/satra/projects/mdd/set1/MBcache/f36e3d5d99f7c4a9bb70e2494ed7340b/OASIS-30_Atropos_template_to_MNI152_affine.txt (type = MatrixOffsetTransformBase)
 stdout 2015-03-07T01:29:21.599079:  2. /om/scratch/Mon/satra/mdd/metaflow/50207/antsct/antsCT_TemplateToSubject0Warp.nii.gz (type = DisplacementFieldTransform)
 stdout 2015-03-07T01:29:21.599079:  3. /om/scratch/Mon/satra/mdd/metaflow/50207/antsct/antsCT_TemplateToSubject1GenericAffine.mat (type = AffineTransform)

the template being used in antsct is nick's adni template. shouldn't there be a transform linking nick's adni template to the oasis-30-atropos template?

here is the complete antsCorticalThickness call and parameter log:

Runtime info
------------

* command : antsCorticalThickness.sh -a /om/scratch/Mon/satra/mdd/metaflow/50207/copy_header/50207_anatomical_hea
derFix.nii.gz -m /om/scratch/Mon/satra/mdd/metaflow/50207/antsct/T_template0_BrainCerebellumProbabilityMask.nii.g
z -e /om/user/satra/projects/mdd/scripts/templates/T_template0.nii.gz -d 3 -f /om/user/satra/projects/mdd/scripts
/templates/T_template0_BrainCerebellumExractionMask.nii.gz -s nii.gz -o antsCT_ -p nipype_priors/BrainSegmentatio
nPrior%02d.nii.gz -t /om/user/satra/projects/mdd/scripts/templates/T_template0_BrainCerebellum.nii.gz
* duration : 38415.91558
* hostname : node005.cm.cluster

Terminal output  

 stdout 2015-02-22T00:56:54.864568:WM[4]  
 stdout 2015-02-22T00:56:54.869143:WM 4
 stdout 2015-02-22T00:56:54.888290:The output directory "antsCT_" does not exist. Making it.  
 stderr 2015-02-22T00:56:54.896795:
 stderr 2015-02-22T00:56:54.896795:  
 stderr 2015-02-22T00:56:54.896795:
 stderr 2015-02-22T00:56:54.896795:
 stderr 2015-02-22T00:56:54.896795:
 stderr 2015-02-22T00:56:54.896795:      anatomical image        = /om/scratch/Mon/satra/mdd/metaflow/50207/copy_
header/50207_anatomical_headerFix.nii.gz
 stderr 2015-02-22T00:56:54.896795:      brain template          = /om/user/satra/projects/mdd/scripts/templates/
T_template0.nii.gz
 stderr 2015-02-22T00:56:54.896795:      convergence             = [45,0.0,10]
 stderr 2015-02-22T00:56:54.896795:      debug mode              = 0
 stderr 2015-02-22T00:56:54.896795:      extraction prior        = /om/scratch/Mon/satra/mdd/metaflow/50207/antsc
t/T_template0_BrainCerebellumProbabilityMask.nii.gz
 stderr 2015-02-22T00:56:54.896795:      extraction reg. mask    = /om/user/satra/projects/mdd/scripts/templates/
T_template0_BrainCerebellumExractionMask.nii.gz
 stderr 2015-02-22T00:56:54.896795:      float precision         = 0
 stderr 2015-02-22T00:56:54.896795:      gradient step size      = 0.025
 stderr 2015-02-22T00:56:54.896795:      image dimension         = 3
 stderr 2015-02-22T00:56:54.896795:      max iterations          = 100x100x70x20
 stderr 2015-02-22T00:56:54.896795:      metric                  = CC[fixedImage,movingImage,1,4]
 stderr 2015-02-22T00:56:54.896795:      output image suffix     = nii.gz
 stderr 2015-02-22T00:56:54.896795:      output prefix           = antsCT_
 stderr 2015-02-22T00:56:54.896795:      prior combinations      = WM[4]
 stderr 2015-02-22T00:56:54.896795:      registration template   = /om/user/satra/projects/mdd/scripts/templates/
T_template0_BrainCerebellum.nii.gz
 stderr 2015-02-22T00:56:54.896795:      regularization          = 
 stderr 2015-02-22T00:56:54.896795:      run quick               = 0
 stderr 2015-02-22T00:56:54.896795:      segmentation prior      = nipype_priors/BrainSegmentationPrior%02d.nii.g
z
 stderr 2015-02-22T00:56:54.896795:      smoothing sigma         = 1.5
 stderr 2015-02-22T00:56:54.896795:      thickness prior         = 10
 stderr 2015-02-22T00:56:54.896795:      transformation          = SyN[0.1,3,0]
 stderr 2015-02-22T00:56:54.896795:      use random seeding      = 1

[binarybottle]

Satra --

Issue 1: The --all option doesn't result in fundus extraction, so unless you've explicitly called --fundi, they won't be generated. Did you call --fundi?

Issue 2: Thank you so much for this detailed account! I will try to think clearly after an hour's sleep (app launch imminent). As you rightly point out when referring to the chain of commands, antsApplyTransforms is the step to focus on. Since the correct labels are in the joint fusion atlas, that leaves the transforms, which you point out as suspicious: "shouldn't there be a transform linking nick's adni template to the oasis-30-atropos template?" I am sure you are right. This is a command Nick Tustison used on ADNI data using his ADNI template that you are using:

antsApplyTransforms -d 3 -i 20080416_1407261284011361922076945140808101991208323033126ADNI912INNOMEDSTUDYs002a1001_BrainSegmentation.nii.gz -o test.nii.gz -r ~/Data/Public/ADNI/Templates/Normal/T_template0.nii.gz -t 20080416_1407261284011361922076945140808101991208323033126ADNI912INNOMEDSTUDYs002a1001_SubjectToTemplate1Warp.nii.gz -t 20080416_1407261284011361922076945140808101991208323033126ADNI912INNOMEDSTUDYs002a1001_SubjectToTemplate0GenericAffine.mat -n NearestNeighbor

If I am thinking clearly, then the right way to do this would be to have the antsCorticalThickness.sh arguments -e and -t point to the same (ADNI) template in MNI space, or to follow up the command with the application of such a transform. This is because mindboggle expects one to use a template in MNI space where the joint fusion atlas resides.

Looking back, I am embarrassed to have sent you an incomplete command in a 5/6/14 email:

Satra -- Here is the antsCorticalThickness.sh command you should run with Nick's ADNI template in $Template_path: antsCorticalThickness.sh -d 3 -n 3 -w 0.25 \ -a IMAGE \ -o PREFIX \ -e ${Template_path}/T_template0.nii.gz \ -t ${Template_path}/T_template0_BrainCerebellum.nii.gz \ -m ${Template_path}/T_template0_BrainCerebellumProbabilityMask.nii.gz \ -f ${Template_path}/OASIS-30_Atropos_template/T_template0_BrainCerebellumExtractionMask.nii.gz \ -p ${Template_path}/Priors2/priors%d.nii.gz

This was a blunder on my part, because a couple of months earlier, I wrote to Nick regarding this very question:

"If you've already run [the longitudinal 3T ADNI] all through FS and antsCT with the [ADNI] template, then all that would be left would be to run them through Mindboggle, yes? Or, since you probably didn't run it with -e and -t pointing to the same template in MNI space, an extra transform would need to be computed, yes?"

to which Nick agreed. Did we correct this in your Mindboggle runs of the ADNI data? Otherwise, we would have gotten empty volume outputs, yes?

To come back to the present, why are you using the ADNI template as opposed to the OASIS-30 Atropos template for the MDD data? Because Nick's template is constructed from a more representative (older) age group of healthy people?

If the above is correct, then I see five ways to proceed:

1. Just use the unaffected surface measures.
2. Re-run mindboggle to do #1 from mindboggle -h: --no_transforms no transforms (antsCorticalThickness.sh run without -t, for example), so no volume outputs, and no data registered to MNI152 space
3. Re-run antsCT with -e and -t options and re-run mindboggle.
4. Transform the antsCT output data and re-run mindboggle.
5. Revise mindboggle to accept such transforms.

What would you recommend?

[satra]

@binarybottle - in the ants command above why are there two paths for templates? it looks like the -f option uses OASIS_atropos template? should i just use the OASIS atropos template for everything?

how do i download the OASIS atropos template? i see one one synapse, but i don't want to download a file at a time. is there a zip file somewhere?

also is it different from this oasis template?

http://figshare.com/articles/ANTs_ANTsR_Brain_Templates/915436

once you let me know where i can download the OASIS template, i'm just going to rerun with the OASIS template for the mdd and sad data. too much of a time pressure right now to get this down. my talk is in a week and a half and i have a bunch of other analyses to do.

[binarybottle]

As you confirmed, the OASIS Atropos templates on mindboggle's site and on figshare.com are the same. I just added a clear section at the top of the Mindboggle-101 webpage:

http://www.mindboggle.info/data.html#mindboggle-software-data

[binarybottle]

Satra -- regarding your original message: "- the csv files are all strings (i.e. requires me to look and figure out what's a number, a label, and a list). if you are going to store numbers and lists, this should be json

• some of the csv files are not formatted consistently or properly, they have spaces with numbers"

I understand why json is a good fit for specifying the data types, but I don't know how many people in medical image analysis or statistics deal with json files. Everybody understands and can deal with csv tables. Three options going forward:

1. generate both tables and json files
2. separate list entries (spectra, moments, positions) from tables of numbers
3. fix the csv tables so that they are very clear

[satra]

    if do_fundi:
NameError: name 'do_fundi' is not defined
Return code: 1

[satra]

do_fundi should be False rather than commented out.

[satra]

now these files have all zeros: tables/left_surface/[sulcus|label]_shapes.csv

[binarybottle]

Yeesh -- clearly I should never quickly debug after an hour's sleep! I just set do_fundi = False by default.

[satra]

@binarybottle -- any thoughts on why these are all zero?

now these files have all zeros: tables/left_surface/[sulcus|label]_shapes.csv

[binarybottle]

I assumed that the bug I introduced (commenting out do_fundi) threw a monkey wrench into the works, so just to be sure I am starting clean, I am reinstalling everything from scratch right now and will run a test once ANTs is finished installing...

[satra]

thanks!

[binarybottle]

As I wrote to you outside of GIthub, I realized that when I recently moved the mindboggle.info website to Github pages, I also moved the files that are accessed and cached by mindboggle. Fixing now...

[binarybottle]

You're going to get a laugh out of this, Satra. Remember those references to 'hemi' above? We needed them after all! I put them back in and now we have volume and surface labels restored.

[satra]

still 0s in the surface tables.

i deleted my work directory, cache directory and output directory and reran mindboggle based on current master. i don't think you pushed your changes.

[binarybottle]

After all that, I didn't push? I need to be more pushy!

(pushed and bushed)

[ryanhammonds]

I am also seeing all 0s for the frontmarginal sulcus row in sulcus_shapes.csv for all subjects.

I am using version 1.3.8 in a singularity image. Freesurfer v6.0.0 was previously ran outside the container, while ants and mindboggle were ran from within the container. This happened in independent datasets.