Closed Simsso closed 5 years ago
Great notebook, @FlorianPfisterer.
Here is my review on the vq_layer
function definition.
scores = tf.reduce_sum(tf.square(tf.expand_dims(x, 2) - tf.transpose(embed_space)), axis=1)
Update: okay, got it. You can save the transpose when expanding on axis=1
.
stop_gradient
over here; I forgot that.
vq_loss = tf.reduce_mean(tf.norm(tf.stop_gradient(x) - chosen_embeddings, ord='euclidean'))
commit_loss = BETA * tf.reduce_mean(tf.norm(x - tf.stop_gradient(chosen_embeddings), ord='euclidean'))
It's a very good idea to add it, even though I'm not sure whether we're going to train our conv layers. But the production VQ layer version should have it!
This loss term is similar to the VQ-VAE paper
Well, that's because my experiments where based on the paper 😆. We're not going to train the conv layers, but it still adds a gradient flow that might be relevant for another VQ-layer further upstream.
This PR contains a vector quantization (VQ) prototype that was developed and tested in four Jupyter notebooks.
The code is not production-ready and only there for the sake of experimenting (it is located in
/experiments
). The well-tested and functionally extended version can now be developed based on this prototype (#51). Merging of this PR resolves #25.The work can be found here (please review the four files +
README.md
):