search

NVlabs / NVAE

The Official PyTorch Implementation of "NVAE: A Deep Hierarchical Variational Autoencoder" (NeurIPS 2020 spotlight paper)
https://arxiv.org/abs/2007.03898
Other
999 stars 163 forks source link

Generating Encoder Output from Images #15

Closed marcosterland closed 3 years ago

marcosterland commented 3 years ago

Hi! I would like to explore the latent variables for given images, in order to cluster images based on their latent space representations. Which output do I have to take to get the variables? Do I have to take z after applying normalizing flows, so around model.py#L366?

Thanks for your help! Marc

PS: Great paper, and thanks for the helpful reference implementation!

arash-vahdat commented 3 years ago

Hi Marc,

Sorry for my slow reply. The line you pointed out samples from z for the very top latent variable group. For the remaining groups, you can use this line: model.py#L400.

One worry about clustering using stochastic z's I have is that most of these variables are not active and they have zero-mean unit-variance distribution. I am afraid that this stochasticity will hurt your clustering results.

I may suggest either i) sample several z's per image and use the mean of these samples or ii) use mean for each random variable that is predicted in the output of encoder here and here.

best of luck!

marcosterland commented 3 years ago

Thanks for the detailed answer, Arash! I'll test it soon and let you know about the results!

SChoi97 commented 3 years ago

Hi @marcosterland I was wondering if you managed to obtain the latent embeddings? If you did, how were they and how were the clustering results? I am very curious to know!

marcosterland commented 3 years ago

Hi @SChoi97 , the latent space embeddings of the NVAE ended up too high-dimensional for my needs. When changing the parameters to reduce the dimensionality, the model wasn't really capable of reconstructing my images. But I started working on a simplified version, which showed nice clusterings. If you perform the clustering on the different hierarchy levels, the groups cluster as expected. I hope I can publish my code & results this summer.

SChoi97 commented 3 years ago

Hey @marcosterland thanks for the response. That sounds promising, is there any chance you could let me know how much you simplified the model? That would be incredibly helpful! If not, I could wait until your results are published :)

azagajewski commented 5 months ago

Hey @marcosterland, I've been wondering if you ever got anywhere with the encoder outputs? I'm also looking at using NVAE for understanding microscopy data and curious which latent variables you tapped to get clusterings :)