ADVI diagnostic

[avehtari]

Summary:

Add PSIS diagnostic for ADVI .

Description:

This paper proposes PSIS diagnostic for ADVI: Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman (2018). Yes, but Did It Work?: Evaluating Variational Inference https://arxiv.org/abs/1802.02538

Currently ADVI samples N draws from the approximate distribution and outputs those. In addition lp column is inserted, but it has all 0's (it seems stansummary requires lp column).

To follow the current design with stansummary, what we propose is

• [x] after ADVI finishes, compute lp and lg (log density of of the approximation) in stan/src/stan/variational/advi.hpp

• [x] output lp and lg along with parameter sample (for lp provide values instead of 0's and add lg column). Something needs to be added also in stan/src/stan/services/experimental/advi/meanfield.hpp

• [ ] add algorithms for computing khat and neff in services

• [ ] Given the output we can compute diagnostic afterwards, e.g., using stansummary for CmdStan

• for simplicity we would implement this first for CmdStan (needs separate issue), ie, modify stansummary to output diagnostics cmdstan/src/cmdstan/stansummary.cpp

Note that initially we start just computing the diagnostics, but eventually we can use diagnostics to improve the algorithm.

[avehtari]

@yao-yl is working on this, but I can't assign this issue for him

[yao-yl]

I do not know why I cannot get assigned or self-assigned. But yes, I am working on this. I have implemented this in cmdstan and will try to add it in rstan.

[bob-carpenter]

I think you need to be added as a collaborator to be assigned---I think @syclik and @seantalts have the permissions to do that for you.

[seantalts]

Weird! I'll look into it.

[drezap]

@yao-yl I was thinking about assigning myself, but it appears you've already implemented this:

I have implemented this in cmdstan

We can do separate PR's for each interface.

Want to drop me the code and I can go ahead and integrate it and write appropriate tests?

[avehtari]

I just updated the issue. 2 first things are in Stan. Next thing would be to implement khat and n_eff in services.

[drezap]

To compute khat I'm using Zhang and Stephens section 4?

[bob-carpenter]

The reference I know for ADVI eval with k-hat is Yao et al..

[avehtari]

You can use the R code as a reference implementation https://github.com/stan-dev/loo/blob/master/R/gpdfit.R It follows Zhang and Stephens , but uses numerically more stable computations plus weakly informative prior.

[drezap]

and then for Neff?

[avehtari]

For Neff see the variance formula which is last equation on page 4 in https://arxiv.org/abs/1507.02646 and for the reference code see https://github.com/stan-dev/rstan/blob/develop/rstan/rstan/R/stanmodel-class.R#L288 lines 288-304