search

cdoersch / vae_tutorial

Caffe code to accompany my Tutorial on Variational Autoencoders
MIT License
500 stars 134 forks source link

VAE loss seems to differ in paper and implementation #2

Closed EmilienDupont closed 7 years ago

EmilienDupont commented 7 years ago

Hi, thanks for a great tutorial on VAEs! I have a quick question about the implementation. In the tutorial, the reconstruction loss is L2 (as I thought it should be). screen shot 2016-12-22 at 12 27 50

However, in the Caffe implementation, there is what seems to be an additional cross entropy reconstruction loss screen shot 2016-12-22 at 12 28 04

What is the purpose of this loss? Or am I missing something?

I realise cross entropy loss if often better for less blurry images, but since we parametrize P(X|z) by a gaussian with mean f(z), I thought the log likelihood should be proportional to ||X - f(z)||^2.

Thank you!

cdoersch commented 7 years ago

This is the definition of the EuclideanLoss layer:

layer {
  name: "loss"
  type: "EuclideanLoss"
  bottom: "decode1neuron"
  bottom: "flatdata"
  top: "l2_error"
  loss_weight: 0
  include {
    phase: TRAIN
  }
}

loss_weight is set to 0 because this isn't actually used in the training objective. I included it just so that you could track the Euclidean loss if you wanted to (or switch it out). This example is actually using the sigmoid cross entropy as the main loss. I discuss the reasons for this in section 4.1 of the tutorial.

I've added a comment to explain this better. Thanks for pointing this out.