search

TuringLang / ParetoSmooth.jl

An implementation of PSIS algorithms in Julia.
http://turinglang.org/ParetoSmooth.jl/
MIT License
19 stars 12 forks source link

Simple `reloo` wrapper #34

Open ParadaCarleton opened 3 years ago

ParadaCarleton commented 3 years ago

Would be nice to have a way to easily refit folds with high Pareto k values. Mentioning @sethaxen because I know ArviZ.jl has this set up; do you know how/if we could get something similar into ParetoSmooth?

sethaxen commented 3 years ago

On the Python side, ArviZ does indeed have reloo for some PPLs. It's implementation is PPL-specific, one approach per supported PPL. ArviZ.jl hasn't added anything like this for Julia PPLs yet. The closest I've gotten is https://github.com/arviz-devs/ArviZ.jl/pull/133, and resampling wasn't on my mind then.

I think the Julian way to handle this would be to use a PPL-agnostic interface, and I'm wondering if AbstractPPL can become that interface. I think what we need are more examples of partial implementations of PPL-agnostic diagnostic/statistic/plotting functions in order to find patterns of what the interface needs. I previously did so for simulation-based calibration, and I wonder if it might be helpful to do the same for reloo.

ParadaCarleton commented 3 years ago

Mmm, I see. Probably something like going through AbstractPPL would be better, yeah.