How can we generate new samples with the model?

[htt210]

Hi Yoon Kim,

In your paper, I can see how SVI helps to reduce to amortization gap but it requires the data x to be available. When VAE is used as a generative model, x is not available so SVI cannot be performed to improve the variational parameters. If we just sample lambda from the prior and feed it to the decoder, my feeling is that the result will be worse than that of VAE because the variational distribution is more different from the prior than before. Does that mean SA-VAE cannot be used as a generative model?

I am also not so sure about the evaluation procedure for text data. Is the input sentence itself served as the target for SVI? Re-estimating the variational parameters at test time will make the model to assign higher probability to every sentence, including sentences which don't make sense. I think a good language model should assign higher probabilities to good sentences and lower probability to bad sentences. I am afraid that SA-VAE is assigning higher probability to every sentence, regardless of the quality.

Can you please help me to address these concerns? Thank you :).

[yoonkim]

Hi sorry I just saw this! Apologies for the late response.

SA-VAE (and its variants) are just for posterior inference. For obtaining random samples from the model you should generate from the prior.

For evaluation, note that the log marginal likelihood does not depend on the variational parameters. The ELBO (which does depend on the variational parameters) is a lower bound to the log marginal likelihood.