confidence intervals for effect size

[pbstark]

Implement inversion of permutation test to find confidence sets for the effect. Start with the assumption that the effect is the same for all pairs. Notes from email of 10/22/2014:

1. Fit a predictive model to all cases, using the covariates but not the treatment dummy. (It might be better to fit just to the control cases, but let's defer that for now.)
2. Use any of the standard matching algorithms to match one or more control case to each treatment case, based on the predictions.
3. The null hypothesis is that, within match groups (pairs or more), the residuals from the prediction are exchangeable. Across match groups, the residuals are independent. This defines the invariance for the permutation test.
4. If the alternative is that treatment increases the response (not necessarily by a constant shift, not necessarily by the same amount in each match group, etc.), a sensible test statistic might be the sum across match groups of the difference within match groups between the mean response with treatment and without treatment. For other alternatives, we might take the sum of absolute values of differences between the means or something else.
5. The distribution of that statistic under permutation within match groups, independent across groups, lets us test. Essentially, we're doing a stratified two-sample permutation test for whether the residuals appear to come from one population or two.
6. Suppose we want to make a confidence interval for the average effect of treatment on the treated. We have some choices. I think the following are all biased towards zero because the predictive model is fitted to all cases. If it were fitted just to the controls--and if it were unbiased for the response of the controls--then the bias would go away.
   1. If we want to posit that the effect of treatment is a shift, we use the standard inversion of permutation tests (but with the stratified permutation). See, e.g., www.stat.berkeley.edu/~stark/Teach/S240/Notes/ch5.htm Search for "estimating the shift." Also see the simulations to estimate the power of the permutation test in http://www.stat.berkeley.edu/~stark/Teach/S240/Notes/ch3.htm What we need is simply to invert those calculations.
   2. Just as easily, we could posit that the effect of treatment is multiplicative, or follows any other functional form.
   3. We could find a fully nonparametric confidence bound if, within each match group, we impose an a priori bound on the effect. (One-sided bounds give one-sided confidence intervals.) Finding the confidence bound involves solving a k-dimensional optimization problem, where k is the number of match groups.

[kellieotto]

Implemented in the permu_CI_mean function. We may want to refine the search/expansion method for numerical precision.