alashworth
commented
5 years ago 
@yao-yl is working on this, but I can't assign this issue for him
Open alashworth opened 5 years ago
alashworth
commented
5 years ago
@yao-yl is working on this, but I can't assign this issue for him
alashworth
commented
5 years ago 
I do not know why I cannot get assigned or self-assigned. But yes, I am working on this. I have implemented this in cmdstan and will try to add it in rstan.
alashworth
commented
5 years ago 
I think you need to be added as a collaborator to be assigned---I think @syclik and @seantalts have the permissions to do that for you.
alashworth
commented
5 years ago 
Summary:
Add PSIS diagnostic for ADVI .
Description:
This paper proposes PSIS diagnostic for ADVI: Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman (2018). Yes, but Did It Work?: Evaluating Variational Inference https://arxiv.org/abs/1802.02538
Currently ADVI samples N draws from the approximate distribution and outputs those. In addition lp column is inserted, but it has all 0's (it seems stansummary requires lp column).
To follow the current design with stansummary, what we propose is
[ ] after ADVI finishes, compute lp and lq (log density of of the approximation) in stan/src/stan/variational/advi.hpp
[ ] output lp and lq along with parameter sample (for lp provide values instead of 0's and add lq column). Something needs to be added also in stan/src/stan/services/experimental/advi/meanfield.hpp
[ ] Given the output we can compute diagnostic afterwards, e.g., using stansummary for CmdStan
[ ] add algorithms for computing khat and neff somewhere in stan/src/stan/variational ?
for simplicity we would implement this first for CmdStan (needs separate issue), ie, modify stansummary to output diagnostics cmdstan/src/cmdstan/stansummary.cpp
Note that initially we start just computing the diagnostics, but eventually we can use diagnostics to improve the algorithm.