All Tau-U Meta Results for A vs. B return a Tau-U value of 1/-1 and an "-Inf" Z Value

[gjones1219]

Hello Juergen,

I hope you're well! I am writing to report a potentially serious bug in the latest version of the "Scan" package. In short, for the Tau-U meta analyses function, the "A vs B" model seems to only return a Tau-U value of 1/-1 and an "-Inf" z value. It returns these values even though some of the individual cases included within the meta-analyses have "A vs. B" values less than 1 or -1 (meaning that there should be no way that the overall Tau-U meta-analysis should have an "Inf" Z score). Furthermore, I just went back and updated some of my prior code to re-run it in the latest version of the scan package (version 0.59) and my results have changed and indeed the A vs. B contrast is now only showing a Tau-U value of 1/-1 and an "Inf" Z score. I would be grateful if you all can look into this as soon possible, as this seems to have a very serious impact on the analyses I am seeking to do at this time (and going forward) and now this test seems to contain some serious inaccuracies. Thank you so much!

[jazznbass]

I don't have the data to replicate the problem but I assume that you have a case in your study where tau is 1.00 for an A vs. B comparison (all B values above all A value). In this case, you would expect a meta-analysis of correlations zo yield a Tau value of 1, z of inf, and p of 0. This is because the correlations have to be Fisher Z transformed before averaging. Which, in the case of Tau = 1, results to Fisher z = Inf. If you do not transform the correlations, you will get confidence intervals outside [-1,1]. So basically, you are aggregating values where one effect strength is infinite and you get an infinite overall result that is above 0 with an infinite high probability.

Try out yourself with the meta package (here for the scan example dataset exampleAB):

meta::metacor(c(1.0000000 ,0.9395973, 0.9664430), c(20, 20, 20))

If you want to aggregate nonetheless (which I do not recommend) you can use the meta package as follows:

meta::metacor(c(1.0000000 ,0.9395973, 0.9664430), c(20, 20, 20), sm = "cor")

(but: the functions of the meta package will remove the 1.00 correlation from the calculation as the reported number of studies for this analysis is k = 2)

I am reluctant to implement this option into the scan package because, to my knowledge, you will get biased results (and sometimes strange CIs). I wonder if it is better to have inaccurate results than no results at all.

@researchtool what are your thoughts on that?

[researchtool]

Hi @gjones1219 and @jazznbass. Thanks, Jürgen, for following up on this so quickly. 😊 I have nothing to add to your explanation of the reasons for the current behavior of the meta-analysis function with Tau values.

I understand the concern about implementing a function that would provide (false) results based on infinite an inf z value and agree that it could lead to biased results and potentially harmful interpretations (by users with less experience than @gjones1219). We all want to ensure the quality and integrity of our research. 💯

However, I also believe that it could be worthwhile to consider implementing this feature as a configurable one, which is disabled by default but can be enabled with a clear warning to the user about the potential implications. It could exclude the specific effect size and throw a warning message. 🤔 As researchers, we often encounter complex and unique datasets, and sometimes we need to try less conventional things. I think it would make sense to have a chance to do it just for educational reasons. ⚡

I believe the key here is to provide an adequate warning and documentation! By doing so, we would maintain the flexibility and adaptability of the scan package while also ensuring that we guide users to make informed decisions based on their unique data analysis needs. 🙂

While we should strive for accuracy, I think that an informed user (or teacher) would sometimes be better off with inaccurate results (and a good warning message) than with no result at all (they would possibly use different software that does not even communicate the problem).

I hope this adds to the discussion, and I look forward to your thoughts. timo

[gjones1219]

Hey Timo and Juergen,

Thanks for your quick replies. I really appreciate it. It poses a serious issue that any singular "Inf" Z value within a Tau-U meta-analysis automatically makes the z value for that entry of the meta-analysis "Inf" and the p-value automatically default to "0". I doubt that many users of this package are looking for that, and instead looking for some way to reasonably balance the results across all the various individual cases.

For instance, see below. If one uses the "meta" package, and runs the following code for the meta analysis with one "Inf" Z value and the rest extremely small Z values... meta::metacor(c(1.0000000 ,0.00001, 0.0001), c(20, 20, 20))

One still gets an "Inf" Z value and a p-value of "0." It feels wrong to base one's Tau-U meta-values at all on a calculation that has clear conceptual limitations and creates serious flaws when interpreting one's results.

Furthermore, this issue is of further concern as previous version) of the Scan package did not have this issue (indeed, I have downgraded to 'Scan' version 0.55 to solve this problem, and now the meta-analysis calculations return results that seem much more accurate). I am using the older version for now to solve this problem.

So I personally would recommend changing back to the old approach as the default, and if you all want to maintain this new approach, making very clear what the potential issues are. As Timo mentioned, some of these interpretations could be quite harmful for someone not acquainted with these kinds of analyses. I will look forward to hearing from you all on the path forward here and thank you for your time and attention again!

[jazznbass]

I think the issue here is more complicated. From a statistical point of view, a Tau of 1.00 is an infinite effect strength. The distribution of correlations is not linear but logistic. A correlation of 1.00 has an infinitely higher effect size than e.g. a correlation of 0.95. Consider you would like to average three weights, 1 kg, 10kg, and something infinitely heavy. What would be the average? Infinite. And no matter what the measurement error of the weights, the results would be infinite with a probability of error of Null.

I am thinking of implementing an approach (which will give a biased result of unknown magnitude) of not Z-transforming the correlations and calculating the standard error with se = sqrt((1 - cor^2)^2/(n - 1)) This is a formula developed by Fisher himself, implemented in software and described in books on meta-analysis. This could definitely give CIs outside [-1/1]. I would not recommend the procedure I implemented before scan <= 0.55. This procedure could lead to much too conservative CIs.

With regard to your example, try: meta::metacor(c(1.0000000 ,0.00001, 0.0001), c(20, 20, 20),sm = "cor") which will give you the results for the approach I described above and may be more in line with your intuition. But it would still remove the 1.00 correlation from the analyses for the reasons I gave at the beginning of this post.

I think, these are the limits of Tau-U, or we are trying things beyond its limits. Basically, a regression approach or a randomization test might be more helpful here.

[gjones1219]

Hi Juergen,

Thanks for this response. While I so appreciate your work on this package and your responses to this inquiry, I do want to also be clear that this issue is a serious design flaw and clear conceptual issue with your package in this version that should be rectified ASAP. I cannot tell if we are yet aligned on that fact, so I want to be unambiguous in my statement about this fact so that this matter gets the attention it deserves and so that others aren't harmed by using the package in its current state.

You provide the example of averaging three items, one with infinite weight, and state that the average weight of these items would be infinite, using that example to justify the output from the Scan package version 0.59 for the Tau-U meta-analysis. The clear issue with this analogy is that while objects with infinite weight are impossible and never occur, Tau-U values of 1/-1 (i.e., single case experiments in which all data points from the B phase are above those from the A phase) are a regularly occurring phenomenon within single case experiments. If your methodology for conducting Tau-U meta-analyses in Scan version 0.59 does not take that fact into account, but instead uses a method that automatically outputs a p-value of "0" whenever a single instance of a Tau-U value of -1/1 occurs, it's a really serious problem that should be rectified as it totally invalidates the output from the tau-U meta analyses in this version.

Although you recommend not using prior versions of the Scan package for Tau-U meta-analyses, that seems to be the only option here; in all publications, I will simply mention that the confidence intervals are conservative (a much preferred alternative to the extremely liberal/automatically "0" p-value estimates in version 0.59). Again, so appreciate your work on this package and for making it available, but it is really important that you all see how serious a conceptual issue this is and that you all take steps to note and correct this soon.

Thank you, gjones1219

[researchtool]

Dear gjones1219, Thank you again for your feedback and this discussion. We genuinely appreciate your time and thoughts on the package and the underlying obstacles in analyzing SC data. Your concerns are well understood. However, it is crucial to note that the problem you have raised is fundamentally a statistical one and not a software issue. Handling Tau-U values of -1/1 (or, more broadly, the handling of infinite effect sizes) is an inherent challenge in statistics, not a flaw/error of scan. As a reader of your paper, I would understand and probably follow your approach. However, it is essential to note that while this decision is understandable (and transparently communicated), it is not necessarily better or more correct.

scan currently has a clear and concise standard for handling such cases, which prevents users from getting misleading results; We cannot recommend using the old version because it provides exactly that: a result that is not accurate. On another note, I want to kindly remind you that the development of the scan package is a project handled by people with full-time commitments outside of this endeavor. We dedicate our free time to build and maintain this software because we believe in its ability to support the scientific community. A core tenet of our work is to build tools that are not only useful but also responsible for preventing the misinterpretation of results, especially when all users may not fully understand the underlying statistical principles.

We truly appreciate your perspective and will discuss this issue with our team and other users. However, it is important to bear in mind that your perspective represents one viewpoint among many that we must consider. To maintain the integrity and usefulness of the scan package for a broad range of users, we have to consider the implications of any changes we make from multiple perspectives and use cases. Rest assured, your feedback will contribute to this ongoing discussion. I prefer a solution that gives informed users the opportunity to choose but also prevents less informed users about the (statistical problem) occurring “under the hood”. Again, thank you for engaging with our project and bringing these concerns to our attention.

Best regards, timo