search

hiwonjoon / tf-vqvae

Tensorflow Implementation of the paper [Neural Discrete Representation Learning](https://arxiv.org/abs/1711.00937) (VQ-VAE).
258 stars 41 forks source link

Hi,is there any guarantee not only a few of embeddings being selected? #2

Open jiqizaisikao opened 6 years ago

jiqizaisikao commented 6 years ago

Hi,i have run your source code,it seems that it works well,but i have one question,for the weights of the embeddings are inited randomly,and in tranning it always select the nearest neighbors,so is there any guarantee that not a few of embeddings always being selected?

hiwonjoon commented 6 years ago

It's a good question. I do not have an answer though since I am not the authors of the paper. In my humble opinion, describing in the conceptual level, embedding is also be trained to maximize interpretability of training examples, so it could become the most informative supports for a dataset.

leao1995 commented 5 years ago

When I run the cifar example, it always selects a fixed subset of the embedding, so the issue indeed exists.

bckim92 commented 5 years ago

Kaiser et al. addressed this issue in their paper (https://arxiv.org/abs/1803.03382) and they called it index collapse.