search

dorianps / LESYMAP

Lesion to Symptom Mapping in R
https://dorianps.github.io/LESYMAP/
Apache License 2.0
32 stars 11 forks source link

Issue with controlling for lesion volume #13

Closed Fatimah-A closed 5 years ago

Fatimah-A commented 5 years ago

Hi Dr. Pustina,

I have an issue with LESYMAP results when I control for lesion volume and use “sccan” method. It seems to give opposite results to the expected one. I have a dataset of 195 subjects (48 affected, binary score of 0 being affected and 1 non-affected) and I ran LESYMAP using these parameters: (method = 'sccan', correctByLesSize = "behavior", minSubjectPerVoxel = "5%")

The results I got were in the area that is reserved in affected subjects when we do a proportional difference map (affected subjects lesion overlap subtracted from non-affected lesion overlap).

LESYMAP stat_img with control for lesion volume (stat_img has only positive values):

d

Proportional difference map showing areas that are more damaged in non-affected subjects:

c

I used the same dataset and ran LESYMAP again without controlling for lesion volume, using these parameters: (method = 'sccan', minSubjectPerVoxel = "5%")

The results I got are as expected from the literature and similar to the proportional difference map.

LESYMAP stat_img without control for lesion volume (stat_img has only negative values):

b

Proportional difference map showing areas that are more damaged in affected subjects:

a

Thank you! Fatimah

dorianps commented 5 years ago

Thanks for reporting it here Fatimah.

First, please note that SCCAN has not been tested very well with lesion correction.

If I am understanding correctly the problem, the locations are correct but the sign of the SCCAN weights is flipped, right?

Is this the same identical analysis? If yes, why is the difference map, which has nothing to do with analyses, shifting between the two runs?

Also, can you please paste the info.txt file for each of the analysis. The voxel weights in SCCAN may be flipped if the the behavioral weights are also flipped or the canonical correlation values are negative. I thought I checked these scenarios but who knows what regression out lesion size from behavior is doing to these relationships.

Fatimah-A commented 5 years ago

Thanks for your prompt response!

Well, only one of the locations is correct (without controlling for lesion volume). Here is the issue rephrased. The significant results flip the direction of the findings when I control for lesion volume . That is, when running the analysis without controlling for lesion volume there are findings in the midline of the cerebellum which we know to be the region associated with the deficit of interest (the voxel values are negative).
When we attempt to control for lesion volume. The findings are in a different region of the brain and the statistically significant results are positive voxel values, being associated with a protective effect relative to the deficit of interest. We are fairly confident in the results that do not control for lesion volume because we have seen the same result using mass univariate statistics or a simple proportional subtraction.

The two analyses are identical - the only difference is the correction for lesion volume. The proportional difference map is a non-statistical way of identifying areas that are more damaged in cases compared to controls by subtracting their lesion overlap from the lesion overlap of controls. This difference map has positive values showing areas that are more damaged in controls compared to cases (similar to LESYMAP when controlling for lesion volume) and negative values showing regions that are more damaged in cases compared to controls (similar to LESYMAP without controlling for lesion volume).

I ran these before the last update to LESYMAP that fixed the Info.txt file issue - Here is what I have copied from R for LEASYMAP analysis with control for lesion volume: Found optimal sparsenes 0.042 (CV corr=0.18 p=0.012) Calling SCCAN with: Components = 1 Use ranks = 1 Sparseness = 0.042 Cluster threshold = 150 Smooth sigma = 0.4 Iterations = 20 maxBased = FALSE directionalSCCAN = TRUE Permutations = 0

Without control for lesion volume: Found optimal sparsenes 0.017 (CV corr=0.342 p=9.73e-07) Calling SCCAN with: Components = 1 Use ranks = 1 Sparseness = 0.017 Cluster threshold = 150 Smooth sigma = 0.4 Iterations = 20 maxBased = FALSE directionalSCCAN = TRUE Permutations = 0

Please let me know if this includes what you need from the text file or do I need to re-run it with the updated version?

Thank you! Fatimah

dorianps commented 5 years ago

I am looking for these two values which are not there:

  output$sccan.eig2 = sccan$eig2
  output$sccan.ccasummary = sccan$ccasummary

I am not entirely sure what is going on, I would suggest you stick without corrections for now.

It is also a good practice to try every analysis with a mass-univariate counterpart. In this case it would be method='t-test' or method='BMfast', with alternative='two.sided'. Check how do the univariate results with lesion volume correction compare to SCCAN.

Processing tip: If you want to rerun SCCAN to obtain eig2 and ccasummary, you can skip all the optimization part and just specify the sparseness you found above, i.e., sparseness=0.042.

Dorian

Fatimah-A commented 5 years ago

Here you go!

Info.txt - sccan without control for lesion volume:

LesymapVersion: 0.0.0.9202 method: sccan correctByLesSize: none multipleComparison: none pThreshold: 0.05 flipSign: FALSE minSubjectPerVoxel: 5% nperm: 0 binaryCheck: FALSE noPatch: TRUE showInfo: TRUE sparseness: 0.017 Subjects: 195 patchinfo: not computed Range statistic: -1 0 Suprathreshold voxels: 413 Number of Clusters: 1 Cluster sizes (voxels): 413 Cluster sizes (mm3): 413 sccan.eig2: 0.166240952902011 optimalSparseness: 0.017 CVcorrelation.stat: 0.341746760351677 CVcorrelation.pval: 1.01055045225741e-06

Info.txt - sccan with control for lesion volume:

LesymapVersion: 0.0.0.9202 method: sccan correctByLesSize: behavior multipleComparison: none pThreshold: 0.05 flipSign: FALSE minSubjectPerVoxel: 5% nperm: 0 binaryCheck: FALSE noPatch: TRUE showInfo: TRUE sparseness: 0.042 Subjects: 195 patchinfo: not computed Range statistic: 0 1 Suprathreshold voxels: 265 Number of Clusters: 1 Cluster sizes (voxels): 265 Cluster sizes (mm3): 265 sccan.eig2: 0.124036415769844 optimalSparseness: 0.042 CVcorrelation.stat: 0.158231733255666 CVcorrelation.pval: 0.0271525915035215

Info.txt - BMfast with control for lesion volume:

LesymapVersion: 0.0.0.9202 method: BMfast correctByLesSize: behavior multipleComparison: fdr pThreshold: 0.05 flipSign: FALSE minSubjectPerVoxel: 5% nperm: 1000 binaryCheck: FALSE noPatch: FALSE showInfo: TRUE alternative: two.sided Subjects: 195 patchinfo: Computed Total voxels in mask: 55842 Unique voxel patches: 27883 Average patch size: 2 voxels Unique single voxels: 21313 Range statistic: -21.92 0 Suprathreshold voxels: 99 Number of Clusters: 1 Cluster sizes (voxels): 78 Cluster sizes (mm3): 78

Info.txt - BMfast without control for lesion volume:

LesymapVersion: 0.0.0.9202 method: BMfast correctByLesSize: none multipleComparison: fdr pThreshold: 0.05 flipSign: FALSE minSubjectPerVoxel: 5% nperm: 1000 binaryCheck: FALSE noPatch: FALSE showInfo: TRUE alternative: two.sided Subjects: 195 patchinfo: Computed Total voxels in mask: 55842 Unique voxel patches: 27883 Average patch size: 2 voxels Unique single voxels: 21313 Range statistic: -8.04 6.6 Suprathreshold voxels: 4829 Number of Clusters: 2 Cluster sizes (voxels): 4694 78 Cluster sizes (mm3): 4694 78

Here is a screenshot of the stat_img files. It seems like the results are very similar using "sccan" or "BMfast" methods. The "red" mask is the stat_img resulting from "BMfast" method with control for lesion volume, while "blue" is using "sccan" with control for lesion volume.

bmfast

This is a screenshot of "BMfast" in "blue" without controlling for lesion volume and "sccan" in "red" also without control for lesion volume.

bmfast_sccan

It seems like correcting for lesion volume regardless of the method gives unexpected and inaccurate results of where it should be. Please let me know if this clarifies the problem or if there is anything else you want me to try with my data.

Thank you! Fatimah

dorianps commented 5 years ago

The general similarity with univariate is not surprising, the advantage of SCCAN should be in spatial accuracy. Make sure the signs of the relationships are also similar between SCCAN and BMfast (you may want to check BMfast with multipleComparison='FWERperm' if you include these findings in the paper, it is more reliable than fdr in my experience).

For some reason I don't see the expected sccan.ccasummary variable which would help understand whether the SCCAN weights are indeed flipped after lesion size correction. From what I can see the weights are flipped:

Info.txt - sccan without control for lesion volume: this should have negative relationship with behavior

Range statistic: -1 0 # voxel weights
sccan.eig2: 0.166240952902011 # behavior weight

Info.txt - sccan with control for lesion volume: this should have positive relationship with behavior

Range statistic: 0 1  # voxel weights
sccan.eig2: 0.124036415769844  # behavior weight

Note that SCCAN results are more robust without lesion volume correction (CV.correlation = 0.342) even though less voxels predict behavior (i.e., sparsness 0.017 vs 0.042). This seems a good indication that the lesion volume correction is distorting the relationships in a counter-intuitive way (normally, lesion volume correction lead to more specific/narrow results, while we see the oppositve). Again, this is just brainstorming, I haven't checked systematically what happens with other data.

Fyi, a recent paper by Thye and Mirman shows that SCCAN results can track the functional area very well without a need for lesion volume correction (at least for some deficits, see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6205357/, see Fig 6).