Surprisingly small p-values when comparing two levels of covariate in the presence of only zeroes

[svteichman]

Hi @ailurophilia, I wanted to open a discussion with you about a surprising result that @adw96 and I have been noticing in a strange edge case in radEmu analyses. I've attached a .Rmd and knitted html in a .zip file that explain this phenomenon, as well as my reasoning for why it might be happening and a suggestion of how to address it. We'd love to get your thoughts on this issue though before we implement anything to address it.

In short, I've been noticing surprisingly large robust score test statistics and small p-values for parameters that compare a baseline level of a category covariate to a second level for a category in which those two levels only have observations with zero counts for that categories. I think of this as an uninteresting signal scientifically, because in our data we are only comparing zeros, yet it leads to the smallest p-values in two different data analyses that I've run. I think that this is happening because of the very small empirical score covariance for the parameter of interest in these settings, which leads to a small robust covariance of the score and a large test statistic.

If you could review this html file in the .zip file when convenient, we look forward to hearing your thoughts!

Thanks! Sarah

zero_comparison.zip

[svteichman]

I also noticed this behavior when I accidentally ran my version of radEmu on a dataset where I hadn't filtered out categories with 0 counts across all samples. These categories had very small p-values and were associated with the only small Storey q-values in the analysis. Let me know if it would be helpful to have the data for this example and I can share it.

I'm planning to add a check to radEmu to stop with an error if any categories have 0 counts across all samples to avoid running into these situations (unlike the case I describe above, this seems like a setting in which we don't want to include these parameters in our model at all).

[ailurophilia]

Hi @svteichman – sorry for my very delayed response! Thanks for catching this and looking into it – really, really appreciate your efforts and insights here (and in general work on radEmu)! This is definitely weird behavior, and I don't have an immediate explanation for why we see this, but my suspicion is it might be some combination of the facts that (1) we use an approximation to the score equations to fit null models, (2) we use an empirical variance estimator (as you thoughtfully explore in the zero comparison script), and (3) our asymptotics assume population means are strictly positive, so if we observe all zeroes, perhaps we are too far from asymptopia, even if n is large. In any case, continuing to give this a think and will check back in if a useful thought occurs. In the meantime, I think flagging situations where this shows up is a great idea. Thanks!

[svteichman]

Closing this because it is addressed in PR #74. Now there is a default option to set very small version of these p-values to NA (and other options about how to address these p-values).