search

casperkaae / parmesan

Variational and semi-supervised neural network toppings for Lasagne
Other
208 stars 31 forks source link

Resampling bernoulli in iw_vae #8

Closed kastnerkyle closed 9 years ago

kastnerkyle commented 9 years ago

The bernoulli resampling is happening every epoch - this matches the IWAE paper but is a bug, since resampling is effectively data augmentation the result cannot be compared to VAE on a fixed binarization (such as that provided by Hugo Larochelle here http://www.dmi.usherb.ca/~larocheh/mlpython/_modules/datasets/binarized_mnist.html).

In particular this line: https://github.com/casperkaae/parmesan/blob/master/examples/iw_vae.py#L289

They mention this issue at the bottom of the github repo https://github.com/yburda/iwae

casperkaae commented 9 years ago

Hi Kyle

Thanks for pointing that out. I agree that the resampling makes the results non comparable to the *NADE results but on other hand directly comparable to the IWAE results by Burda (thats why we choose the resampling procedure).

I think I'll just add a note highlighting the issue and a switch to choose between the binarized dataset provided by Hugo Larochelle and the resampled MNIST dataset.

btw it gives 1-2 NATS to do the resampling every epoch.

kastnerkyle commented 9 years ago

That all sounds nice. Since there are so many different "binarized" MNISTs it will be nice to compare the difficulty in modeling a few different types.

On Sat, Oct 10, 2015 at 9:37 AM, Casper Sønderby notifications@github.com wrote:

Hi Kyle

Thanks for pointing that out. I agree that the resampling makes the results non comparable to the *NADE results but on other hand directly comparable to the IWAE results by Burda (thats why we choose the resampling procedure).

I think I'll just add a note highlighting the issue and a switch to choose between the binarized dataset provided by Hugo Larochelle and the resampled MNIST dataset.

btw it gives 1-2 NATS to do the resampling every epoch.

— Reply to this email directly or view it on GitHub https://github.com/casperkaae/parmesan/issues/8#issuecomment-147089983.

casperkaae commented 9 years ago

Agreed, thats left for future work :)