Lab 2 - Effect Size vs R^2

[ClaudiaHebert]

In Lab 2 I noticed that the model with the largest effect size has a low R^2.

The lab does not ask this but for my own understanding, does this mean that you likely would not go with this model because of the low fit despite the large effect size?

[lecy]

In the impact evaluation context, select the model that minimizes bias in your policy variable (and when possible add controls to maximize statistical power). Selecting the model with the largest effect size would be a dishonest approach (you are tipping the scale to make the program look effective), and R^2 is less interesting to evaluators because you don’t care how much impact all of the things outside of your control influence the outcome (the control variables), you care how much your intervention influences the outcome (the effect size). Least bias in B1 should be your first criteria, precision of that estimate (small standard errors) is second.

[ClaudiaHebert]

Thank you! How do I judge the bias of B1?

[youngjaewon]

Hi Claudia,

We can measure bias, defined as the difference between population parameter and estimate (B1 - β), in the case of data generated directly by researchers as simulated data, because the causal effect of y on x has been set by the researchers. For example, if there is simulated data based on the relationship y=3*x+e, and when regressing y on x, the OLS estimate of x is 2.8, then the bias would be -0.2 (calculated as 2.8 - 3).

In the case of real-world data (data collected from observation, not simulation), bias cannot be measured directly. However, the properties of an estimator (e.g. OLS estimator) can be studied through mathematical proofs or by using simulated data to determine whether the estimator is unbiased or has less bias than other estimators.