How to compare one graph to a group

[carlosoleto]

Hello,

I'm a new user of SPM1D, and I have a fundamental question regarding comparing one single piece of data, one graph, to a group. For example, comparing single GRF data to a group made from a set of GRF data.

I already tried using ttest, after interpolating all graphs to Q=101, and used the single GRF as the y0 argument (spm1d.stats.ttest(Y, y0=None, roi=None), y0 — optional Q-component datum array (default is the null continuum)). The results seem odd since a single GRF, pretty much equal to the mean, still fails to fall within the 0.05 significant level.

When I used ttest2 to compare the whole group with a set of just two graphs (np.array([group[6], group[24]])), I got the expected results.

Should I discard using the t-test for one sample? Does it make sense to create small sets of graphs (original + noisy versions) and use ttest2?

Is there another way to compare a single graph to a group?

Best regards.

[0todd0000]

The results seem odd since a single GRF, pretty much equal to the mean, still fails to fall within the 0.05 significant level.

I don't quite understand this problem. If the datum y0 is quite similar to the sample mean, wouldn't one expect the test to not yield significance?

Does it make sense to create small sets of graphs (original + noisy versions) and use ttest2?

This is not a good idea because it is inappropriate to create artificial data for use in a hypothesis test.

Can you please attach figures depicting the results for the ttest and ttest2 cases?

[0todd0000]

I fear that I may have misinterpreted your main point, so please allow me to re-attempt a response:

1. For the one-sample test ( spm1d.stats.ttest ) you expected there to be no significant effect, but you observed significance.
2. For the two-sample test ( spm1d.stats.ttest2 ) you also expected there to be no significant effect, and as expected you did not observe significance.

Is this interpretation correct?

[carlosoleto]

Hello, Dr. Pataky. I'm sorry if I was not clear. Thanks a lot for helping with this subject.

To provide more context, I want to compare one sample to a group. I have already done this with n-dimensional feature space and the Mahalanobis Distance, which gives a Chi-Squared distribution and a statistical threshold of similarity (I hope I'm not misunderstanding this again).

So, in my current research, I would like to use SPM1D since I judged it important to compare the whole data instead of just some key points.

The analysis that I'm trying to do is to compare simulated jump data (acceleration of CoM) to a sample of captured jumps performed by athletes (acceleration based on GRF).

So, I tried using spm1d.stats.ttest(Y, y0=None, roi=None), y0 — optional Q-component datum array (default is the null continuum), where Y would be my 50 accelerations, and y0 would be jump n6 from the sample. I got the results below:

So, In my interpretation, the spm1d.stats.ttest failed, despite being pretty much close to Y mean.

When I used ttest2 with jumps n6 and n24, I got the results below:

So, ttest2 with two of the original samples, which are close to Y mean, passed.

Looking at the test results, I suppose that using the ttest providing y0 is not working in my case. The test does not allow much deviation from the mean, and it fails very easily (sorry for the poor choice of words).

Should I still use spm1d in this context? Maybe, if I create a set of different simulations (maybe different initial guesses or starting positions) then I could use ttest2 the right way?

Thanks a lot for the help again.

Best regards.

[0todd0000]

Thank you very much for the figures and clarifications. I believe that these results can be explained in terms of sample size and power as follows:

• The one-sample results are over-powered due to a large sample size
• The two-sample results are under-powered due to both (i) small sample size, and (ii) group size imbalance

These interpretations stem from general hypothesis testing principles:

• Larger sample sizes increase a test's power.

Here "power" refers to statistical power, which the probability that a test will identify a true effect. A sample size of 50 is much more likely to identify a true, small effect than a smaller sample size of 10, for example. If you find the identified effects in the one-sample test to be small from a scientific or clinical perspective, it implies that the sample size is too large and that the test is overpowered. A pre-experiment power analysis, wherein the hypothesized effect(s) of interest are explicitly identified, would help to identify an appropriate sample size for identifying those effects.

• Small sample sizes and group size imbalances reduce a test's power

For the two-sample test the group sizes are 2 and 50, representing a large imbalance. Generally as imbalance grows a test's power declines. The more serious issue is that one group has just 2 observations. With just 2 observations the sample mean and variance can deviate substantially from true population mean and variance. Probability calculations account for this sample mean and variance error in the following manner: as sample size decreases, the critical threshold for statistical significance must increase in order to avoid regarding large sample mean errors as "significant".

[carlosoleto]

Thanks a lot for the answer, Dr. Pataky.

I'll consider your comments when I try to use spm1d again.

Unfortunately, I will need to use other tools in my current research.

Best regards.