Add L-BFGS to Turing - Githubissues

TuringLang / Turing.jl

Bayesian inference with probabilistic programming.

https://turinglang.org

MIT License

2.02k stars 219 forks source link

Add L-BFGS to Turing #421

Closed yebai closed 5 years ago

yebai commented 6 years ago

The Optim.jl package provides an implementation of BFGS, perhaps we can integrate it with Turing?

xukai92 commented 6 years ago

Note: link for LBFGS in Optim.jl - http://julianlsolvers.github.io/Optim.jl/stable/algo/lbfgs/

emilemathieu commented 6 years ago

In what context shall L-BFGS (a memory efficient quasi-Newton method) be added to Turing ? Is Turing now supporting some sort of optimisation paradigm ?

yebai commented 6 years ago

The Flux.jl package has implementations for several popular optimisation algorithms in ML. Perhaps we can consider to re-use them in Turing:

http://fluxml.ai/Flux.jl/stable/training/optimisers.html

@xukai92 @ChrisRackauckas

ChrisRackauckas commented 6 years ago

I believe it just added some higher order methods too @mikeinnes

willtebbutt commented 6 years ago

I completely agree with @yebai that we should try to make use of Flux's optimisers. This is perhaps a useful conversation to have at JuliaCon. It's not clear to me whether making this work would simply be a case of using a Turing.jl VarInfo to construct a Flux.jl param, and then adding a couple of methods to make stuff work with the new optimiser interface, or whether it will be a bit trickier for some reason.

xukai92 commented 6 years ago

The new interface seems to be compatible with what Turing.jl's internals can provide. Though until they adapt a generic AD I would assume their grad may have problem with Distributions?

MikeInnes commented 6 years ago

We don't currently have any gradient definitions for Distributions, but they are easy to add and we'd be very happy to have them in Flux. Let me know if you have any trouble getting that set up; I'll also be around at JuliaCon and would be happy to meet up over this stuff.

willtebbutt commented 6 years ago

I'll also be around at JuliaCon and would be happy to meet up over this stuff.

This would be very helpful.

yebai commented 6 years ago

We don't currently have any gradient definitions for Distributions, but they are easy to add and we'd be very happy to have them in Flux. Let me know if you have any trouble getting that set up; I'll also be around at JuliaCon and would be happy to meet up over this stuff.

I agree with @willtebbutt and @MikeInnes that we should talk at JuliaCon. Turing and Flux together can offer a much stronger support for ML: while Flux focuses more on Neural Networks (NN) models and optimisation based learning, Turing focuses more on Probabilistic Machine Learning (PML) and Bayesian inference.

In particular, I think it would be nice if Turing and Flux can

share AD and optimisation procedures
share GPU and parallel computing libraries
have compatible syntax for model and inference compositions
have example models that combine NN/DL and Bayesian Inference

yebai commented 5 years ago

Closed in favour of https://github.com/TuringLang/Turing.jl/issues/605