0todd0000
commented
8 years ago Hi Bernard,
Thank you for the questions, they're very good but also not very easy to address.
Regarding the hypothetical examples... Allow me to propose another example which might help to resolve the issue.
- Experiment 1: p = 0.045
- Experiment 2: p = 0.055
Would you conclude that Experiment 1 has uncovered a true effect and that Experiment 2 has not?
This conundrum pertains to the fallacy of classical hypothesis testing, and in particular to the binary null hypothesis rejection decision. From a strict classical perspective, and for alpha=0.05, one would reject the null hypothesis for Experiment 1 and would fail to reject the null for Experiment 2, even though the probabilistic difference between p=0.045 and p=0.055 is quite small. From a modern perspective, and in particular from a Bayesian perspective there is effectively no difference between Experiment 1 and Experiment 2. The Bayesian approach accomplishes this by removing both alpha and the step-like binary null hypothesis rejection decision, and replacing them with a smoother, more continuous perspective regarding observed effects. The Bayesian approach is widely regarded as superior to the classical approach, and one reason is exactly the fallacy of the binary null hypothesis rejection decision. Classical hypothesis testing is powerful, but it is unable to yield satisfactory interpretations of results close to alpha.
I would therefore recommend against binary interpretations of small differences like those in the three examples you provide. For the same reason I would also recommend against making conclusions regarding precise temporal windows. Instead I would regard these examples as practically equivalent, and all temporal windows as approximate estimations of a potentially true effect.
Regarding your other points:
because the inference was made on smoothed residuals
Note that the residuals are not smoothed. The residuals are smooth if and only if the original data are smooth. You might actively smooth the original data (e.g. using a Butterworth filter), and this will likely cause the residuals to become smoother, but no additional smoothing is applied to the residuals.
we consider that the scalar field did contribute to the vector field at that time period
I think the modern perspective mentioned above will give you the answer you're looking for: try to avoid binary interpretations of small differences, including whether or not particular vector components contribute to the vector result.
but this should provide grounds for specific discrete time point hypothesis testing.
I don't think smoothing is related directly to the issue of whole-trajectory vs. discrete point analysis. In my opinion the distinction is the hypothesis you make (explicitly or implicitly) before conducting the experiment. If you have an a priori hypothesis regarding a particular point then discrete time point hypothesis testing is perfectly appropriate. If you don't explicitly generate such a hypothesis then in my opinion the a priori hypothesis implicitly pertains to the whole trajectory, and whole-trajectory methods like SPM and FDA may be more suitable.
Todd
bernard-liew
Dear Todd,
I am writing this on the same issue that was touched on previously (https://github.com/0todd0000/spm1d/issues/38). I am inquiring on this due to a second comment from a reviewer of a paper. Let me repose the same hypothetical scenario
**"Say I did a vector field and significance was found in 50% to 90% of gait.
If significance in post hoc at scalar level was found in: e.g 1) 49% to 90% e.g 2) 50% to 91% e.g 3) 49% to 91%
In e.g 1-3, can we say post hoc was significant only in the region where primary analysis was significant? Or if the bounds is exceeded, we should say we cannot interpret the post hoc results."**
In addition to the answer you provided, can it be said that because the inference was made on smoothed residuals, and smoothing of a signal in fact means discrete points no longer exist, a significant scalar field cluster can only either contribute or not contribute to the primary vector field (at the approximate period). To act on the conservative side (avoid Type 2 error), we consider that the scalar field did contribute to the vector field at that time period, but this should provide grounds for specific discrete time point hypothesis testing.
Would you agree with the above comment or maybe you could provide a more refined additional explanation?
Really appreciate the help.
Regards, Bernard