stuff to sample/plot

[mjhajharia]

list of transforms to sample:

• augmented-ilr
• hyperspherical_angular
• logistic
• probit

plots currently:

• cdf and density for ess/leapfrog
• rmse vs cum. leapfrog steps

any other transforms or plots we want to add? do we want something with ess bulk/tail separately or anything

cc: @sethaxen @spinkney

[sethaxen]

• I think it makes sense to compute ESS for both bulk and tail. If the tail reveals nothing useful it can go in the appendix.
• I commented this on the paper, but by CDF, do you mean empirical CDF (ECDF) or CDF of the KDE?
• How is RMSE computed?
• @bob-carpenter suggested in #9 computing the fraction of transitions where the Hessian is negative (I think) definite. This could be computed on a per-chain basis or for the combination of all draws across all chains. Is that something we still would like to do? Not sure if this should be a plot or just a table. I wonder if there are other properties of the Hessian that would be worth looking at and might inform whether the transform would make it hard for HMC to adapt a metric, like the condition number mentioned after Eq 9 of https://arxiv.org/pdf/1905.11916.pdf. (try as I might, I haven't been able to grok that paper)

[bob-carpenter]

Yes, I was suggesting we report fraction of negative-definite Hessians (sorry---I can never keep signs straight---it's the negative inverse Hessian that looks like covariance). But then I also suggested reporting condition number. So that could be a histogram. So maybe histogram plus fraction negative definite?

[bob-carpenter]

Also, RMSE is computed w.r.t. known true answers gathered heavily thinned HMC runs yielding 10K draws.

[mjhajharia]

Yes, I was suggesting we report fraction of negative-definite Hessians (sorry---I can never keep signs straight---it's the negative inverse Hessian that looks like covariance). But then I also suggested reporting condition number. So that could be a histogram. So maybe histogram plus fraction negative definite?

sounds good i'll make a PR

[mjhajharia]

• I think it makes sense to compute ESS for both bulk and tail. If the tail reveals nothing useful it can go in the appendix.
• I commented this on the paper, but by CDF, do you mean empirical CDF (ECDF) or CDF of the KDE?
• How is RMSE computed?
• @bob-carpenter suggested in Geometry of transforms #9 computing the fraction of transitions where the Hessian is negative (I think) definite. This could be computed on a per-chain basis or for the combination of all draws across all chains. Is that something we still would like to do? Not sure if this should be a plot or just a table. I wonder if there are other properties of the Hessian that would be worth looking at and might inform whether the transform would make it hard for HMC to adapt a metric, like the condition number mentioned after Eq 9 of https://arxiv.org/pdf/1905.11916.pdf. (try as I might, I haven't been able to grok that paper)

cdf of kde

but i can get ecdf too (statsmodels.api.ecdf) if that's preferred i guess

[mjhajharia]

• @bob-carpenter suggested in Geometry of transforms #9 computing the fraction of transitions where the Hessian is negative (I think) definite. This could be computed on a per-chain basis or for the combination of all draws across all chains. Is that something we still would like to do? Not sure if this should be a plot or just a table. I wonder if there are other properties of the Hessian that would be worth looking at and might inform whether the transform would make it hard for HMC to adapt a metric, like the condition number mentioned after Eq 9 of https://arxiv.org/pdf/1905.11916.pdf. (try as I might, I haven't been able to grok that paper)

alright i'll check that out, and yes actually had a conversation with yuling about this - he also suggested using sampler divergences like wasserstein distance. which has quite some connections to convexity as well, trying to write it all down in a coherent way, will get back about this soon

[sethaxen]

cdf of kde

but i can get ecdf too (statsmodels.api.ecdf) if that's preferred i guess

I think that makes sense. Unlike KDE or a histogram, the ECDF (at sufficiently high resolution) contains all of the information of the original sample. There's no benefit to passing through a KDE first.