search

pymc-devs / pymc

Bayesian Modeling and Probabilistic Programming in Python
https://docs.pymc.io/
Other
8.67k stars 2k forks source link

What needs to be done for a release? #1104

Closed springcoil closed 8 years ago

springcoil commented 8 years ago

At PyData London @twiecki and I chatted about what needed to be done for a release, and @jonsedar presented a nice advanced tutorial on it. As I understand it there's just a few things left before a release candidate can be released.

Am I missing something major?

springcoil commented 8 years ago

@fonnesbeck @jsalvatier @taku-y you all may have opinions too...

twiecki commented 8 years ago

The other big missing piece is consistent shape handling (e.g. https://github.com/pymc-devs/pymc3/issues/1079). I talked a bunch with @brandonwillard who has an idea for a refactoring that might break some things. So we'd rather do such a change before 3.0. @brandonwillard anything to add?

fonnesbeck commented 8 years ago

A few things come to mind:

twiecki commented 8 years ago

@fonnesbeck do we have a stalling NUTS example? I wonder if the ADVI init PR https://github.com/pymc-devs/pymc3/pull/1088 helps with that.

taku-y commented 8 years ago

@fonnesbeck Does deterministics mean the variational parameter in the original (not transformed) space? I think it is difficult to transform the standard deviation of the variational posterior in the transformed space back to the original space, because in the original space the variational distribution is no longer than Gaussian. Sorry if I misunderstand what you said.

And should we need a function to draw samples from variational posterior of ADVI? I wrote such a function and applied to the Bayesian neural network example. Unfortunately, in this example, variational posterior failed to fit data (https://gist.github.com/taku-y/6fabb9b905a6639019369ef418960bd2), possibly due to that the mean field approximation is too simple.

fonnesbeck commented 8 years ago

@taku-y the reported estimates from fitting with ADVI currently report only the stochastics, some of which are transformed to the real line, and hence not meaningful to users (we have actually hidden these variables from the trace in the MCMC samplers), whereas the deterministics are often of interest, and it would be helpful if their means and variances were reported automatically.

twiecki commented 8 years ago

@taku-y I definitely think such a sampling function would be useful. Ideally it returns a familiar Trace object. In your example, it's interesting that it doesn't work too well. I think the @stan-dev team uses some more tweaks as well, like a grid-search to optimize the eps parameter and something else to tune the learning rate. Have you looked at these?

brandonwillard commented 8 years ago

@twiecki I can try to get a branch with those changes up this week.

twiecki commented 8 years ago

@brandonwillard :+1:

twiecki commented 8 years ago

@taku-y The other thing that would be great for ADVI is a short tutorial in the form of an IPython NB with some basic info / references.

I agree that the retransformations will be difficult.

taku-y commented 8 years ago

@fonnesbeck Thanks for your comment. I understood what you said.

Instead of getting means/stds of deterministic variables, I'm trying to make a sampling function supporting deterministic variables. But I could not put drawn samples into a trace object. The notebook including the sampling function (sample_vp) is here (https://gist.github.com/taku-y/d768a34f9d3f985bab818669a0106bb7). Could you tell me how to solve the problem?

taku-y commented 8 years ago

@twiecki I will try to refine my Gaussian mixture example; add info of ADVI and references.

taku-y commented 8 years ago

@twiecki Thanks for telling me grid search of hyperparameters in STAN. Parameter tuning may solve the problem. Another option is taking more complex form of variational posterior as done by normalizing flows (http://arxiv.org/abs/1505.05770) and Hamiltonian variational inference (http://jmlr.org/proceedings/papers/v37/salimans15.pdf). Although it is out of scope of this issue, these techniques are interesting and may be useful.

twiecki commented 8 years ago

@taku-y here is the note from @akucukelbir on grid-search: https://github.com/pymc-devs/pymc3/pull/775#issuecomment-204059736

I see you already started work on the normalizing flows, does it work?

taku-y commented 8 years ago

@twiecki Thanks for pointing the code for grid-search. It's helpful for implementation in PyMC3.

I have implemented the normalizing flow but it has only single transform and did not seem show any effect so not confirmed it works. I will stack normalizing flows, say 40, as described in the paper.

taku-y commented 8 years ago

I fixed the problem on recording samples drawn (https://gist.github.com/taku-y/d768a34f9d3f985bab818669a0106bb7). After checking the sampling function on some other models (simple Gaussian and Bayesian NN), I will send a PR.

twiecki commented 8 years ago

@taku-y that looks very promising!

taku-y commented 8 years ago

I fixed a bug in the sampling function and the bayesian NN result looks great (https://gist.github.com/taku-y/6fabb9b905a6639019369ef418960bd2). I think it's almost ready to merge though should add a test.

twiecki commented 8 years ago

@taku-y Wow, that looks great. Very encouraging that mini-batch ADVI works so well in this case. Unit-test would certainly be great but looking forward to the PR.

springcoil commented 8 years ago

How are we doing on all of this, I've not had time to do much lately other than merging other PR's. Am I correct in appreciating that the ADVI stuff is pretty much fixed up and so we're nearly there?

Eh2406 commented 8 years ago

HI, as a random person from the internet. Let me second the "ADVI to be better documented" Just heard about ADVI, and I am no expert. So I don't know what "this is a mean-field approximation" means in practise? More generally what can I do with ADVI? What can I not do with ADVI? Are there ways to use ADVI in conjunction with other methods?

taku-y commented 8 years ago

@springcoil Sorry for late to reply. In addition to documents and including deterministics in sampling function for ADVI, I want to add autoencoding variational Bayes for mini-batch ADVI. It allows us to include local latent variables to models so I believe it would enhance the applicability of ADVI. I will send a proposal as an issue.

taku-y commented 8 years ago

@Eh2406 Thanks for your comments. I think your questions are important especially for practitioners. I will answer these questions in the document.

springcoil commented 8 years ago

Hi @taku-y that's really cool I look forward to it. I'll look into the ADVI documentation too - maybe that's a contribution I could make.

taku-y commented 8 years ago

@springcoil I'm working on autoencoding VB on #1236 and #1241. After these PR accepted, I will start to write some documents.

springcoil commented 8 years ago

What needs to be done here? @twiecki and @fonnesbeck what do you think needs to be done for a release? I think the documentation is coming along.

twiecki commented 8 years ago

I think we can release anytime (and should soon), there's always something.

twiecki commented 8 years ago

I think we should just pull the trigger.

On Sep 3, 2016 7:43 PM, "Peadar Coyle" notifications@github.com wrote:

What needs to be done here? @twiecki https://github.com/twiecki and @fonnesbeck https://github.com/fonnesbeck what do you think needs to be done for a release?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244559981, or mute the thread https://github.com/notifications/unsubscribe-auth/AApJmHfnnqtm_fyM1a0KNTK9tiJ8YQGcks5qmbHTgaJpZM4Ie77t .

springcoil commented 8 years ago

Agreed. Pull the trigger

On 4 Sep 2016 7:45 AM, "Thomas Wiecki" notifications@github.com wrote:

I think we should just pull the trigger.

On Sep 3, 2016 7:43 PM, "Peadar Coyle" notifications@github.com wrote:

What needs to be done here? @twiecki https://github.com/twiecki and @fonnesbeck https://github.com/fonnesbeck what do you think needs to be done for a release?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244559981, or mute the thread https://github.com/notifications/unsubscribe-auth/AApJmHfnnqtm_ fyM1a0KNTK9tiJ8YQGcks5qmbHTgaJpZM4Ie77t .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244587207, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8DiP2C4w4li7Z5U07eXmqNID7uRXAUks5qmmkjgaJpZM4Ie77t .

springcoil commented 8 years ago

Ok let's put a date down - we can fight over this later :) I propose 6th of September. Is that too early? What needs to be changed for a 'release' do we just need to update something on PyPI?

fonnesbeck commented 8 years ago

Well, I was hoping to have the multivariate shape issue and the stalling behavior for NUTS sampling to be solved before a release, but I am happy to set those aside if everyone else is keen.

I will go over the outstanding issues today and see if anything jumps out. We should definitely give master a good PEP8 compliance check and general linting before tagging a release.

springcoil commented 8 years ago

Hi Chris, I think while those are great - pushing for a release now with all the momentum we've had would be great.

I don't think multivariate shape issue is a deal breaker.

Definitely +1 on the PEP8 stuff

On Mon, Sep 5, 2016 at 5:29 PM, Chris Fonnesbeck notifications@github.com wrote:

Well, I was hoping to have the multivariate shape issue and the stalling behavior for NUTS sampling to be solved before a release, but I am happy to set those aside if everyone else is keen.

I will go over the outstanding issues today and see if anything jumps out. We should definitely give master a good PEP8 compliance check and general linting before tagging a release.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244782232, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8DiCekLswK8uYbSUzkMWspRMQqUaKoks5qnEN7gaJpZM4Ie77t .

Peadar Coyle Skype: springcoilarch www.twitter.com/springcoil peadarcoyle.wordpress.com

springcoil commented 8 years ago

I think assuming @twiecki nails the PEP8 stuff today, which I think he will, all the other things like test fixes, and other fixes by myself last night mean I think we're ready for a release candidate. I'd love to announce this today at PyData London Meetup.

jsalvatier commented 8 years ago

I just want to cheer you guys on! Great job!

springcoil commented 8 years ago

Hey John - thanks man - you of course can't be ignored in this effort :)

On Tue, Sep 6, 2016 at 4:30 PM, John Salvatier notifications@github.com wrote:

I just want to cheer you guys on! Great job!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pymc-devs/pymc3/issues/1104#issuecomment-244989413, or mute the thread https://github.com/notifications/unsubscribe-auth/AA8DiCjBOduBoP0EJRcludU9kehIaNRcks5qnYcSgaJpZM4Ie77t .

Peadar Coyle Skype: springcoilarch www.twitter.com/springcoil peadarcoyle.wordpress.com

springcoil commented 8 years ago

Now we have a release candidate done. I'll close this.