search

TuringLang / ParetoSmooth.jl

An implementation of PSIS algorithms in Julia.
http://turinglang.org/ParetoSmooth.jl/
MIT License
19 stars 12 forks source link

Tutorials+Documenter.jl #28

Closed ParadaCarleton closed 3 years ago

ParadaCarleton commented 3 years ago

We need to add tutorials, and the nice-looking documentation/tutorials that most packages get by using Documenter.jl.

ParadaCarleton commented 3 years ago

@goedman Do you want to work on this? I think this fits together perfectly with StatisticalRethinking.jl; the tutorial examples can probably be taken straight from Statistical Rethinking.

goedman commented 3 years ago

Hi @ParadaCarleton Unfortunately I really don't have the time to do that properly at the moment.

I certainly will redo the examples in chapters 7 & 8 while I'm refactoring the SR code base over the next month or so. Whoever writes the tutorial can draw from those examples.

I might use ParetoSmooth slightly differently as I want it to adhere to the StatisticalRethinking terminology and result presentation (e.g. comparison with WAIC and addition of the parameter penalty).

From this perspective the workflow with the older LooCompare struct felt a bit more convenient as just having the estimates and pointwise (vs the PsisLoo objects) and dropping the simple Array based method is a bit counter intuitive to me, at least coming from Stan. But that's just a preference issue I guess and might not be a valid argument for Turing.

ParadaCarleton commented 3 years ago

Hi @ParadaCarleton Unfortunately I really don't have the time to do that properly at the moment.

I certainly will redo the examples in chapters 7 & 8 while I'm refactoring the SR code base over the next month or so. Whoever writes the tutorial can draw from those examples.

I might use ParetoSmooth slightly differently as I want it to adhere to the StatisticalRethinking terminology and result presentation (e.g. comparison with WAIC and addition of the parameter penalty).

From this perspective the workflow with the older LooCompare struct felt a bit more convenient as just having the estimates and pointwise (vs the PsisLoo objects) and dropping the simple Array based method is a bit counter intuitive to me, at least coming from Stan. But that's just a preference issue I guess and might not be a valid argument for Turing.

That's actually fairly interesting -- I actually switched the LooCompare struct to align more with the API used in the loo package, so it would be more familiar to R users. I also switched out the array-based command so the frontend could be interface-neutral: the idea is to make it so that PSIS-LOO objects can be generated using any of Stan, Turing, Gen, Soss, or just a generic array, and then we can just work with PSIS-LOO objects without having to deal with writing comparison methods for every one of these input methods.

The list of individual PSIS-LOO objects is no longer stored inside the model comparison struct because that requires copying over the objects, which can be very computationally intensive and wasteful of memory when the sample size is large.

As for the terminology, I completely hear you; I like the Statistical Rethinking-style names better than what I'm using at the moment. The terms I'm using have been chosen like this since I plan to add support for other utility functions in the future, in which case names like elpd wouldn't accurately represent the objects. I'm trying to work around this problem by implementing some kind of double-labeling scheme that lets users index into arrays with the ML-like terminology I'm using right now, but will also display more familiar variable names like p_eff when the utility function chosen is the standard Bayes loss.

I completely understand that you're busy though; thank you so much for all your hard work!

ParadaCarleton commented 3 years ago

Thanks a bunch to Mohamed for helping with this!