flag_tfce

[normanF]

Dear Armand, first of all, thank you for this impressive work, both the toolbox and the paper (and the thesis respectively).

I'd like to ask you what would happen if we turn off 'flag_tfce' option during the call to ept_tfce? Would the function then return sort of pixel-wise t_max permutation? And if so, are there other switches, where I could obtain e.g. cluster-mass correction in order to compare the performance of the different approaches as you did in your paper?

Best regards Norman

[Mensen]

Hi Norman,

Thanks for the kind words! I still use it every other day for various analyses so while the coding of the toolbox could use a little redoing, the principles still hold.

You are correct, with flag_tfce = false; it returns a classical maximum permutation test for each channel. Note these are not pixels as SNPM would do since that involves both the reshaping of electrode locations onto a 2D surface (and can/will distort what is considered a neighbouring electrode), and also involves the interpolation of data in between those points which is, at best, inefficient use of computation, and, at worst, in some violation of statistical independence.

I used to have the cluster-size and cluster-mass options within the toolbox, but found that in constantly updating the toolbox over the years, I was unmotivated to make sure that statistical options I felt were invalid / insensitive / unspecific wasn't worth the time to also maintain. I could send you the old scripts and if you were really interested I don't think it would take long at all to make sure they were in working order.

However, in adjusting the e and h parameters of TFCE you basically obtain a generalised version of the cluster or size mass approach. For instance, setting the h parameter to 0 indicates that at each threshold level, you add together the cluster sizes with no regard to the intensity... essentially representing the cluster size approach (without the need for the arbitrary cluster forming threshold). If you then also introduced a starting threshold parameter (for TFCE always 0 and maximum is the max T-value), then you would get the equivalent cluster size results for that threshold. Note also that setting e = 0 is therefore basically the classic max-Tvalue approach where no regard is given to the cluster size.

I always find reframing the classical approaches in this way introduces a helpful perspective for them. Often the criticism of TFCE is that it "introduces" two new parameters e and h ... but actually those two parameters are just hidden in the cluster or max-T approach and in those are set to quite silly values. All TFCE really does is make those parameters explicit and in doing so eliminate the need for arbitrary thresholds. The great side effect of doing this more valid approach is that it happens to make the analysis more sensitive to distinct signal types and easier to interpret. Of course when stated in these simple terms it would be hard to make a thesis out of it :)

[normanF]

Hey Armand,

thanks a lot for your answer. Now a couple of questions are clearing up for me! Also the tfce formula appears more intuitive. Makes totally sense that setting either H or E to 0 results in the cluster size or the max-T approach given that we fix the threshold h. I still wonder, however, how clustermass, ie. summing neighboring t-values, might be achieved with the tfce algorithm under a fixed threshold, since the number of neighboring points, e, is multiplied by the (one) t-value, h, that is currently evaluated. Or maybe I still miss something.?

The reason why I asked is that clustermass in your paper appeared to be more sensitive in some cases. For the dataset I currently analyse, I found a significant ITPC condition difference in an electrode, time and frequency of interest. With tfce I aimed at exploring the whole time-frequency-space information. However, nothing was significant anymore, not even the hypothesis driven comparison. So I wondered whether clustermass might be a little more sensitive in this case.

As the dataset has just 32 electrodes and ITPC values appear to be less smoothed in time, I played around with different combinations of E and H, mainly reducing E and increasing the value for H. However, this did not have an effect. I also changed the cluster threshold in the call to > ept_calculateClusters (up to 0.9), but still no change. Then I went on and inspected the results structure of > ept_tfce. The Obs field showed sensibly varying (t-?) values. However both TFCE_Obs and maxTFCE were all zero. The only case I could imagine where this might be true is, when there are no neighbors...which is not true. So I guess I must do something crucially wrong. Here is how I called the function: results_supra = ept_TFCE(data1,data2,elocs,'rsample',supra.fs,'fsample',1,... 'type','d','plots',0,'e_h',e_h,'nperm',nperm,... 'savename',['tfce-supraDiff_',num2str(e_h) '_',date,'.mat'],'chn',ChN,... 'flag_tfce',true); where elocs is an eeglab channel structure and ChN I calculated and checked before. The only option here I don't understand is 'fsample' but it doesn't appear to affect anything.

Any advice would be highly appreciated.

[Mensen]

At the theoretical level, its general true that cluster size/mass approaches are more sensitive to differences, especially to broadly distributed signals. However, this sensitivity depends on choosing the right threshold for that particular signal type (impossible to know a-priori). Moreover, this sensitivity comes at a cost of specificity and false positive rates increasing beyond the desired 0.05 level. I know in research its tempting to ignore increased false positive rates since it means its likely to find something significant.

When testing channels, frequency and time, you are giving random permutations a lot of opportunity to show higher difference values than the observed data since only the max is taken for the empirical distribution. While its not an ideal approach... I would be sympathetic to an approach that narrowed in on certain points of interest that are hypothesis driven (after a large test is run and results are illustrated). Again, its not ideal, and one would think that a strong effect shines above the rest regardless of the amount of data examined.

---

Indeed TFCE_Obs and maxTFCE should really not be 0 so something is going wrong in the calculation unfortunately (although maybe fortunately since the right calculation might yield some significant differences after all). Is the output also all zeros for a test case such as:

test_data = rand(4, 50);
ChN = fliplr(eye(4));
tfce_test = ept_mex_TFCE2D(test_data, ChN, [0.5, 2]);

If so we'll have to do some debugging to see what the problem in the scripts could be. Otherwise there might be something in your particular data structure that we'll need to sort out.

Let me know what you find and we'll try to fix the error.

[normanF]

Thanks again a lot for you input, Armand!

Yes you're right, it's tempting to devalue false positive rates if the test is more sensitive. However, shouldn't be the standard. Here planned contrasts already gave significant results with t-values ~4. Their absence in permutation testing puzzled me and therefore wanted to investigate. And, yes, the output of your code indeed provides all zeros. So I was thinking this might be related to compiling the binaries on my system.? I remembered there were some warning messages:

Building with 'gcc'. Warning: You are using gcc version '5.4.0'. The version of gcc is not supported. The version currently supported with MEX is '4.9.x'. For a list of currently supported compilers see: http://www.mathworks.com/support/compilers/current_release. /path/ept_TFCE-matlab-master/TFCE/Dependencies/ept_mex_TFCE2D.c: In function 'mexFunction': /path/ept_TFCE-matlab-master/TFCE/Dependencies/ept_mex_TFCE2D.c:259:6: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types] dims = mxGetDimensions(prhs[0]); ^ /path/ept_TFCE-matlab-master/TFCE/Dependencies/ept_mex_TFCE2D.c:260:7: warning: assignment from incompatible pointer type [-Wincompatible-pointer-types] dims2 = mxGetDimensions(prhs[1]); ^ /path/ept_TFCE-matlab-master/TFCE/Dependencies/ept_mex_TFCE2D.c:263:37: warning: passing argument 2 of 'mxCreateNumericArray' from incompatible pointer type [-Wincompatible-pointer-types] plhs[0] = mxCreateNumericArray(ndim,dims,mxDOUBLE_CLASS, mxREAL); ^ In file included from /licensepath/software/matlab/9.2/extern/include/mex.h:8:0, from /path/ept_TFCE-matlab-master/TFCE/Dependencies/ept_mex_TFCE2D.c:30: /licensepath/software/matlab/9.2/extern/include/matrix.h:224:34: note: expected 'const size_t {aka const long unsigned int }' but argument is of type 'const int *'

define mxCreateNumericArray_730 mxCreateNumericArray

^ /licensepath/software/matlab/9.2/extern/include/matrix.h:782:1: note: in expansion of macro 'mxCreateNumericArray_730' mxCreateNumericArray_730(size_t ndim, const size_t *dims, mxClassID classid, mxComplexity flag); ^ MEX completed successfully.

As mex completed successfully, I thought it's save to go on...

I have now overwritten the compliled binaries by the the ones from your repo and tfce_test is not all zero anymore. Currently waiting for the true dataset comparisons and will let you know.

Best, Norman

[normanF]

Dear Armand, completely forgot to report back, I'm so sorry! But the actual reason is a good one. After using your versions of compiled mex files, tfce calculation started working smoothly and I could go on with investigating very exciting effects. Now, I don't want to miss your tool in the future! Kudos Norman

[Mensen]

That's wonderful to hear! Definitely let me know if you run into any more issues or have further questions!