Sigma-scale Plot

[brandonwillard]

We need a plot comparing HS and sqlasso in terms of the scale parameter, $\sigma$ (c.f. Figure 2 in [1]).

[1]: Polson, Nicholas G, and James G Scott. “Shrink Globally, Act Locally: Sparse Bayesian Regularization and Prediction.” Bayesian Statistics 9 (2010): 501–38.

[DattaHub]

I have created the HS bimodal plot, need to show that the sqrt-lasso doesn't suffer from bimodality. Here is the MWE:

set.seed(123)
beta = 20; sigmasq = 1
X = matrix(c(1,1),byrow=T)
y = X%*%beta + rnorm(2)
#library(horseshoe)

res <- horseshoe(y, X, method.tau = "truncatedCauchy",
                       method.sigma = "Jeffreys",
                 burn = 1000, nmc = 5000, alpha = 0.05)
#library(ggplot2)
samples.hs <- data.frame(values = t(unlist(res$BetaSamples)))
ggplot(data=samples.hs,aes(x=values))+
  geom_density(position="identity")

[brandonwillard]

It looks like horseshoe's Jeffery's prior for $\sigma^2$ produces samples from the reciprocal of a Gamma with shape 2.0 and scale around 1/100. This puts the expected value of those samples pretty far out there.

If you replicate the observations, e.g.

X = matrix(rep(c(1, 0, 0, 1), 20),
           ncol=2, 
           byrow=T)

beta = c(20, 20)

y_hat = X %*% beta

(y = drop(y_hat + rnorm(nrow(X))))

then horseshoe produces reasonable results (as expected).