equivalence testing

[PosturalControlLab]

Dear all, I recently started working with this nice package. Is there a way to do equivalence testing with it? I am thinking about something like the 'two one-sided' test. As far as I understand this is not implemented. To be a bit more specific with respect to my problem: I would like to show that an experiment we have done in VR is similar/equivalent to results in real world environments (I compare complex valued vectors across frequency). So I would like to test the Null-Hypothesis that they are different. I tried to set this up with two one-sided tests, but failed to obtain anything reasonable. Any suggestions? Thank you! Lorenz

[0todd0000]

Hi, thank you for this comment.

You are corrects: equivalence tests, including TOST procedures, are not implemented in spm1d. One reason they are not implemented is that effects themselves are 1D, so if the hypothesized effect is non-constant across the 1D domain, then typical equivalence procedures cannot be used.

To get an idea for the complexities related to 1D effects please check these 1D power examples. Like equivalence tests, power analysis requires an effect model, and generally the effect model is non-constant. There are no theoretical solutions for non-constant effects, so numerical analysis is the only way these power problems can be solved.

Could you use a two-tailed two-sample t test? (spm1d.stats.ttest2) This is simple to implement, and the only disadvantage is that hypothesized effects cannot be included.

[PosturalControlLab]

Dear Todd, thank you for this detailed and helpful reply!

I did use a two-tailed 1D Hotellings T² test (I compare complex valued frequency response functions) and it does give me reasonable results. So in general, using statistics to support my conclusions, this does make sense (and will probably also convince reviewers). But, following a typical fisher statistics approach, I believe I should be using an equivalence test.

As far as I understand this now, there is a conceptual problem. You state that typical equivalence procedures cannot be used to show that 1D trajectories in two conditions are identical. Thus, I guess other approaches then fisher statistics need to be used (which probably don't exist yet). Is this correct?

[0todd0000]

Sorry about that, I think "typical procedures cannot be used" was poor wording. Everything about the procedures can be identical, with one major exception: probability calculations. When an effect is geometrically simple (e.g., a scalar offset across the entire time domain), then there is a analytical density function from which probabilities can be calculated. For geometrically complex effects, like those in the power examples at the link above, analytical solutions do not exist, so probabilities must be computed numerically, and nonparametrically, in a Monte Carlo-like simulation.So the same approach can be used, but you would have to calculate probabilities nonparametrically if the hypothesized effect is geometrically complex.

Note that many geometrically simple effects like an offset are not biomechanically plausible in many situations. Consider ground reaction forces during stance: the offset must be zero at the endpoints, because force is zero at the endpoints by definition. The modeled effect would have to be nonlinear to be biomechanically plausible.

[PosturalControlLab]

Thank you for the clarification. Seems I got this quite wrong. I will stick with the non-parametric two-tailed Hotelling's T² test for now. I guess the equivalence tests assuming e.g. a difference in % could be something for a feature request. So I will close the issue here. Thank you again for your input! Lorenz