search

TuringLang / AdvancedVI.jl

Implementation of variational Bayes inference algorithms
http://turinglang.org/AdvancedVI.jl/
MIT License
63 stars 14 forks source link

Minibatches #38

Open dan-sprague opened 1 year ago

dan-sprague commented 1 year ago

Hey guys,

I'm working with a large dataset with a relatively large number of parameters (last-layer approximation for a neural network). Out-of-the-box VI is simply a non-starter here.

To perform parameter updates in mini-batches, is scaling the contribution of the minibatch to the log-likelihood the primary change?

Thanks for your work!

Red-Portal commented 1 year ago

Hi, this is a little bit late for feedback, but the current codebase lacks the proper knobs to do doubly stochastic VI. You'll pretty much have to implement things from scratch. I have a personal codebase that does it, so please let me know if you need any assistance.

dan-sprague commented 7 months ago

Yes I started looking into the codebase and when trying to make adjustments realized that the way things were implemented made a scratch implementation necessary. Decided to use Pyro since it supports this. Not sure how much active development this library still gets but it would be a tremendously useful feature for practical applications.

Red-Portal commented 7 months ago

@spragud2 Hi, we're currently working on rewriting AdvancedVI entirely. Minibatching support is expected to come very soon after that. Hopefully by the end of this year or earlier next year.