Covariate Selection: include imbalanced covariates?

[YinghuiZhouu]

Mutz et al 2019

Main focus:

Find whether there are statistical evidence to support the argument: “Covariates may be included because their distribution is unbalanced across treatment groups.” (Mutz et al., 2019, p. 34)

Main argument:

• “Absolutely any result can be obtained when there are sufficiently many covariates (Pham 2016, Theorem 2.1).” ( p. 34)
• “In typical experimental settings, BT&A can easily lead to a false boost that pushes the p-value across a threshold such as 0.05 or 0.01”
• “including covariates in clean experiments for reasons other than efficiency opens the floodgates to coaxing weak coefficients across the critical p-value threshold.” (p. 41)
• the reduction of Type I error when weeding out unlucky draws is shown to be small compared to the introduction of Type II error.
• “To summarize the results of this section, it is almost always possible to obtain a false positive by adding covariates indiscriminately, requiring a pool of only a few dozen. If one limits the cheating to adding a single covariate, the Type I error, nominally set at 5%, jumps to between 6% (N = 400) and 8.5% (N = 50).”

Theorem

Let $V(X) = C(X, X)$ denote empirical variance. The estimators

$$ \hat{\beta}_0 = \frac{C(X, Y)}{V(X)} $$

$$ \hat{\beta} = \frac{C(X, Y)V(Z) - C(X, Z)C(Y, Z)}{V(X)V(Z) - C(X, Z)^2} $$

are the result of regressing $Y$ on $X$ alone or on both $X$ and $Z$, respectively.

Suppose that the truth is given by the linear model

$$ Y = \mu + \beta X + \gamma Z + \theta \xi, $$

where $\xi_j$ are independent mean zero unit variance noise terms. Despite the omission of $Z$ from the first model, both $\hat{\beta}_0$ and $\hat{\beta}$ are unconditionally unbiased estimators of $\beta$. The variance of $Y$ explained by $Z$ is $\gamma^2E(Z^2)$, while the variance unexplained by either $X$ or $Z$ is the variance $\theta^2$ of the noise term. The ratio

$$ T := \frac{\gamma^2 V(Z)}{\theta^2} $$

is a parameter of the model measuring the portion of variance of $Y$ explained by $Z$. The following result is proved in the online appendix.

Theorem 1. Given the $X$ and $Z$ variables, let $r^2 = \frac{C(X, Z)^2}{V(X)V(Z)}$. Then $$ E{\ast}(\hat{\beta} - \beta)^2 \leq E{\ast}(\hat{\beta}0 - \beta)^2 $$ if and only if $T > N^{-1}(1 - r^2)$, where $E{\ast}$ denotes conditional expectation with respect to the $X$ and $Z$ variables.

“r2 is known before treatment”

“This result says that one should be less inclined to include Z, not more, if you see that Z is imbalanced (r2 is large)”

An intuitive explanation is that collinearity between the treatment and the covariate introduces uncertainty as to which is responsible for any variation predicted in the dependent variable. The threshold, it should be noted, is small, and minimizing the variance of the estimator is far from the only goal. Theorem 1 does not dictate what variables to include, but it does imply that a failed balance test is not a good reason for including a variable in the analysis.

Theorem 2. Consider two estimators of a treatment effect, one the simple difference of means estimator and one post-stratified by a covariate taking the values 0 and 1. The post-stratified estimator will have a lower MSE than the simple estimator if and only if the ratio of variance in the dependent variable predicted by the covariate to the unpredicted variance is greater than

$$ \frac{1}{n} [ab(a + b) + cd(c + d)](a + b)(c + d) \div abcd(a + b + c + d). $$

Here, $a$, $b$, $c$, and $d$ are the respective sizes of the groups of controls with covariate value 0, controls with covariate value 1, treated subjects with covariate value 0, and treated subjects with covariate value 1.

The threshold is somewhat complicated, but it is minimized when the group sizes are equal, and it always grows when the distribution of the covariate within a treatment group becomes more imbalanced. This means that a failed balance test is not a good reason to perform a stratified analysis. If a balance test fails, the threshold for stratification to improve efficiency increases rather than decreases.

Adam Zelizer's R program

Setup:

• Equal treatment and control units (250 each)
• Number of simulation: 1000

Main results

[YinghuiZhouu]

New direction to show (in Chapter 4): Don:

• [ ] Conditional on significant covariate imbalance for a prognostic covariate such as pre-test, the sampling distribution is no longer centered on the true ATE. (This argument comes up in FEDAI where we discuss ex ante vs. ex post unbiasedness.) We should show that using DD for this example.
• [ ] In other words, if we know that a pre-test seems to favor the treatment group, the sampling distribution conditional on pretest_treat > pretest_control is much larger than the true ATE. Saying that imbalance might not be a good criterion for reducing standard errors misses the point; the conditional sampling distribution is biased and presumably the mean-squared error is larger than if we control for the imbalanced covariate.

As I understand:

• [ ] show that in those random assignments that produced a treatment group with higher average pre-scores than the control group, the difference-in-means estimator generates a higher average estimate. And we can recover the true ATE by controlling for the pre-scores. (the corresponding discussion on FEDAI is in section 4.3, pages 108-109. ) We can use the same data in Table 4.1 to show this.

[YinghuiZhouu]

Jiawei: I would first consider what constitutes a better test for imbalance, or what might be a more meaningful null hypothesis for the imbalance test (similar to the parallel trend pre-test in Difference-in-Differences analysis)? I personally agree with the null hypothesis that covariates are imbalanced rather than balanced, as argued in this paper: https://onlinelibrary.wiley.com/doi/full/10.1111/ajps.12387.

[YinghuiZhouu]

To-do from meeting May 10th:

• [x] varying the proportion of truly prognostic covariates. Taking Adam set, but expanding it and contracting it instead of having it be a fixed set.
• [ ] related to p.36 "If we have reason to believe the coefficient of Z is zero or very small, it does little good to demand lack of bias in the event that Z is a strong predictor. This can be seen most clearly in the case where Z and Y are independent. It is intuitively clear that the estimator βˆ0 that ignores Z should perform better than the estimator βˆ that uses irrelevant information.". It's not that clear. burning up degrees of freedom that should make the standard errors larger. But you're also reducing the sum of squared residuals even by chance because you've got another variable. And the rate at which those two things counterbalance each other, it's not totally obvious. It would be useful to do a simple check to see how much precision is lost when include irrelevant regressors. Guess degrees of freedom with irrelevant regressors is really only a concern when you have a very small study.
• [x] 1. changing the share of irrelevant regressors
• [ ] 2. how much of a penalty is there for including relevant regressors
• [ ] 3. building on on Zelier's simulation but tweaking it a little bit: ex post, given that you saw that there's evidence of the imbalance, what is the mean squared error? Of the corrected versus the uncorrected? Basically, the difference in means versus the regression adjusted estimator. So, it's a little different from what what Adam did because he said, Just do a balance test. And if there's significance, just go ahead with the adjustment. What's the combination of unbalance in the treatment and significant on Y.
• [ ] 4. LASSO bias
• [ ] 5. run a regression of y on the covariance and then you pick the queries that are significant. And you put those into the final regression. How biased is that?
• [ ] 6. pair matched experiments. Imagine I have a lot of pairs, and I'm randomly assigning observations within pairs. But a lot of the pairs could be not very good prognostic. But let's suppose I just analyze the data in a kind of standard way. I'm going to include a dummy variable for every pair. So imagine I start with 100 observations, I've got 50 pairs, a burning of half the degrees of freedom in my study. But what if those pairs turned out to be not very prognostic? Or like some are prognostic but most are not? would it really be worthwhile to burn up half the observations? Because effectively, a pair match study is essentially creating a dataset that's half the size and just analyzing the differences.
• [ ] 7. LASSO balanced versus imbalanced.