search

TuringLang / ParetoSmooth.jl

An implementation of PSIS algorithms in Julia.
http://turinglang.org/ParetoSmooth.jl/
MIT License
19 stars 12 forks source link

Splitting LOO into its own package #41

Open sethaxen opened 3 years ago

sethaxen commented 3 years ago

PSIS is a useful algorithm in its own right whenever one is doing importance sampling, and so it makes sense for it to be in a lightweight package separate from LOO. Because this package is named ParetoSmooth.jl, this seems to be the sensible home for PSIS.

I propose the creation of a new package named LeaveOneOut.jl or LeaveOneOutCV.jl that would contain the (re)loo functionality currently in this package and have ParetoSmooth.jl as a dependency. This would also allow this package to be a more lightweight dependency.

Stand-alone loo/psis packages are already on the ArviZ.jl roadmap, and it would be nice to join efforts on these: https://github.com/arviz-devs/ArviZ.jl/issues/129

devmotion commented 3 years ago

Isn't it possible to leverage some of the existing CV implementations, e.g. in MLJ?

(Edit: Just to be clear, this question is orthogonal to whether/how the package should be split)

sethaxen commented 3 years ago

I haven't looked at the CV implementations in MLJ, but LOO doesn't actually do CV. It does approximate CV using importance sampling (hence the PSIS implementation). I'd guess MLJ is doing something different.

ParadaCarleton commented 3 years ago

Isn't it possible to leverage some of the existing CV implementations, e.g. in MLJ?

(Edit: Just to be clear, this question is orthogonal to whether/how the package should be split)

I think we can piggyback off of them a bit; I'm actually making a PR right now that implements some MLJ interfaces. However, I think MLJ's evaluation functions do exact cross-validation, given an algorithm. Unless MLJ provides some way to do cross-validation, given some black-box refitting function? I'm very unfamiliar with MLJ, so I'd love to hear if this is possible.