Reporting P > alpha for one-way anova

[Liamdhughes]

Hi guys,

This is a follow-up to issue #222, in which it is mentioned that P should not be reported when P > alpha. However, i was wondering how to write and justify this in a research article and whether that alone would be the only statistic to report.

I am comparing four conditions of limbs with a relatively low number of participants (n=8), and I have run several one-way spm1d ANOVAs, with none of the results crossing alpha. I need to report this in the manuscript in a way that is clear to the reader and reviewer. I was hoping for some advice or answers on the following points:

Has there, to anybody's knowledge, been anyone before who has reported only P > alpha in their research?

Are there any other report statistics that may be useful to report when P > alpha in time-series data?

Do you have any suggestions on how I could justify writing “Statistical Parametric Mapping revealed no significance, with no p levels alpha in all variables analysed; therefore, no p levels are reported” to avoid scrutiny from reviewers?

Many thanks,

Liam liam.hughes@ntu.ac.uk

[0todd0000]

I think it would be fine to report the results that way, but it may be more convincing to reviewers if you plot results similar to the figure below to show how close or how far away from the critical threshold the observed effects were.

My interpretation of issue #222 is that focus was on point-specific p-values (rather than a p-value for a whole, single test) but re-reading the issue I see that my interpretation may have been incorrect. Regardless, (a) it is indeed possible to calculate both types of p-values, but (b) these p-values generally become inaccurate as p gets large.

If you would like to calculate and report the p-value for the maximum F-value in an ANOVA effect, you can calculate it as shown in the script below.

import numpy as np
import matplotlib.pyplot as plt
import spm1d
from spm1d import rft1d

np.random.seed(1)
y     = rft1d.randn1d(24, 101, 20)
A     = np.array([0]*8 + [1]*8 + [2]*8)
f     = spm1d.stats.anova1(y, A, equal_var=True).inference(0.05)
p     = rft1d.f.sf(f.z.max(), f.df, f.Q, f.fwhm)

print( f'f_max = {f.z.max():.5}, p={p:.5}'  )

# plot
plt.close('all')
f.plot()
f.plot_threshold_label(bbox=dict(facecolor='w'))
plt.xlabel('Time (%)', size=20)
plt.show()

The output is:

f_max = 1.4662, p=0.82369

So you could report this p-value of 0.824, provided you report it with the caveat that large p-values are generally inaccurate.

[Liamdhughes]

That's great; thank you for the detailed answer, Todd.