search

TengMCing / lineup_residual_diagnostics

A Plot is Worth a Thousand Tests: Assessing Residual Diagnostics with the Lineup Protocol
1 stars 0 forks source link

JCGS reviewer 1: (4) formal tests give us definite answers #14

Closed TengMCing closed 11 months ago

TengMCing commented 1 year ago

Reviewer 1: "It is true that formal diagnostic tests tend to be too sensitive and yet we don't have a definite sense of how much divergence from the assumed model would cause a significant problem. How much non-normality in the residuals is acceptable? We don't really know. Formal tests give us definite answers about violations of assumptions which we might find hard to ignore but diagnostic plots give us license to sweep inconvenient features under the carpet. Is there really an advantage to the diagnostic plot in this case other than the opportunity to hand-wave about our answer?"

TengMCing commented 1 year ago

The reviewer believed that in some cases we may intentionally ignore inconvenient features found in diagnostics plots (pretending there is no issue), but formal tests will give us definite answers about whether some assumptions have been violated which forces us to pay attention.

This would be the case if we do not have the lineup protocol. With the lineup protocol, we can assess whether certain visual features actually exist. It is still a formal hypothesis test. It just has different properties than conventional tests.

dicook commented 1 year ago

When the diagnostic plots are embedded in a lineup of nulls, it does have the advantage of providing a definitive answer, and protects against the subjectiveness of hand-waving. We are advocating that you should always calibrate the diagnostic plot with a comparison with null plots. We have strengthened this point of view by adding a sentence in the introduction, in addition to what is provided in the Conclusions.

(Is there a paper to refer the reader to?)