Open ParadaCarleton opened 3 years ago
On the Python side, ArviZ does indeed have reloo for some PPLs. It's implementation is PPL-specific, one approach per supported PPL. ArviZ.jl hasn't added anything like this for Julia PPLs yet. The closest I've gotten is https://github.com/arviz-devs/ArviZ.jl/pull/133, and resampling wasn't on my mind then.
I think the Julian way to handle this would be to use a PPL-agnostic interface, and I'm wondering if AbstractPPL can become that interface. I think what we need are more examples of partial implementations of PPL-agnostic diagnostic/statistic/plotting functions in order to find patterns of what the interface needs. I previously did so for simulation-based calibration, and I wonder if it might be helpful to do the same for reloo.
Mmm, I see. Probably something like going through AbstractPPL would be better, yeah.
Would be nice to have a way to easily refit folds with high Pareto k values. Mentioning @sethaxen because I know ArviZ.jl has this set up; do you know how/if we could get something similar into ParetoSmooth?