VQ-Layer Improvement Ideas

[Simsso]

Let's use this issue to collect some ideas to improve our performance.

• Post-training vector removal: After training is completed, there might be some vectors in the embedding space which are not being used a single time, when feeding all training and validation data through the model. These vectors should be removed, with the goal of improving robustness on unseen activations. There is no advantage in keeping them.
  Extending this idea, we could remove embedding vectors based on the impact that their removal has on the model performance. No matter what the access count is.
• Fusing: Joining two adjacent embedding vectors into one, if samples of the same labels use them.
• PCA on activations (VQ-PCAE): Reduce the dimensionality by performing a PCA on the activations and apply the VQ to the result.
• VQ-VAE between layers. Could be easily implemented with a 1x1 convolution, followed by vq_layer, and another 1x1 conv.
• Spatial quantization: Currently, we are quantizing along the depth (channels), it might be interesting to work with image patches (related: transpose, patch extraction, depth to space, reconstruction)
• Embedding label diversity: Keep track of how many different labels use the same embedding vector. The more vec_size grows, the lower this number. For vec_size equal to the entire activation vector size, only one label can be assigned to every vector.
• Adversarial example training

[Simsso]

Plot validation accuracy vs. number of dropped dimensions (through PCA).

[Simsso]

Effect of PCA compression (pasted here for the wiki page):

[FlorianPfisterer]

Amazing graph! 😍