alashworth
commented
5 years ago 
I completely agree. Same for L-BFGS --- it should be abstracted from the application as much as possible for re-use.
Is there SGD in ADVI now? There shouldn't be given that there's no way for users to run it yet --- things should live on branches until they're ready to go.
We made the mistake of putting in higher-order autodiff and discrete sampling infrastructure before either were ready and it's just been a huge burden.
Thanks for the refs!
- Bob
On Jul 31, 2015, at 3:23 PM, Dustin Tran notifications@github.com wrote:
It would be worth having something generic for all things related to stochastic gradient descent, to be separated from variational inference itself. E.g., a sgd class to have different stochastic gradient methods available, a learning rate class for testing various learning rates, a subsampling class, etc. This will eventually be necessary as we start working on more research tracks, e.g., Mandt and Blei (2014), Theis and Hoffman (2015), Tran et al. (2015).
It should also be applicable for computing the penalized MLE, so that the optimization interface of Stan also has SGD available for users.
— Reply to this email directly or view it on GitHub.
It would be worth having something generic for all things related to stochastic approximations, to be separated from variational inference itself. E.g., a sgd class to have different stochastic gradient methods available, a learning rate class for testing various learning rates, a subsampling class, etc. This will eventually be necessary as we start working on more research tracks, e.g., Mandt and Blei (2014), Theis and Hoffman (2015), Tran et al. (2015).
It should also be applicable for computing the penalized MLE, so that the optimization interface of Stan also has SGD available for users.