search

seongminp / transformers-into-vaes

Code for "Finetuning Pretrained Transformers into Variational Autoencoders"
https://aclanthology.org/2021.insights-1.5
36 stars 11 forks source link

Confusion about the mutual information metric #3

Open smolPixel opened 2 years ago

smolPixel commented 2 years ago

Hello, thank you very much for making the code available. I'm confused about the mutual information math, more specifically about the line

E_{q(z|x)}log(q(z|x)) = -0.5nzlog(2\pi) - 0.5(1+logvar).sum(-1) neg_entropy = (-0.5 nz math.log(2 math.pi) - 0.5 (1 + logv).sum(-1)).mean()

When I derive it, it gives me neg_entropy = (-0.5 nz math.log(2 math.pi) - 0.5 (logv).sum(-1)).mean()

So I think I must have made a mistake somewhere? Thank you

seongminp commented 2 years ago

Hello. I followed the calculations provided in previous research.

(The link is not the original.. If you search for calc_mi, several VAE repos come up.)

smolPixel commented 2 years ago

Thank you for your fast answer, I will try to see from previous research. Would you have by any chance any reference which show the derivation? From what I can see everyone cite Hoffman and Johnson in their paper, which calculate MI between the code and the index by subtracting terms to the ELBO, but calculate in github the MI between X and Z by analytical solution+estimation of the aggregate posterior, following, I think https://github.com/jxhe/vae-lagging-encoder/blob/master/modules/encoders/encoder.py. I'm not sure my math is right or if the two ways are equivalent but I can't find any papers that show the derivation. Thank you very much.