Unintentional decay of embeddings towards 0?

zalandoresearch / pytorch-vq-vae

PyTorch implementation of VQ-VAE by Aäron van den Oord et al.

MIT License

534 stars 101 forks source link

Open samedii opened 4 years ago

samedii commented 4 years ago

By updating all embeddings regardless of if they are being used you are decaying them towards 0. Is this intended?

I tried removing the decay but it seems to decrease perplexity.