Let's use this issue to collect some ideas to improve our performance.
Post-training vector removal: After training is completed, there might be some vectors in the embedding space which are not being used a single time, when feeding all training and validation data through the model. These vectors should be removed, with the goal of improving robustness on unseen activations. There is no advantage in keeping them.
Extending this idea, we could remove embedding vectors based on the impact that their removal has on the model performance. No matter what the access count is.
Fusing: Joining two adjacent embedding vectors into one, if samples of the same labels use them.
PCA on activations (VQ-PCAE): Reduce the dimensionality by performing a PCA on the activations and apply the VQ to the result.
VQ-VAE between layers. Could be easily implemented with a 1x1 convolution, followed by vq_layer, and another 1x1 conv.
Embedding label diversity: Keep track of how many different labels use the same embedding vector. The more vec_size grows, the lower this number. For vec_size equal to the entire activation vector size, only one label can be assigned to every vector.
Let's use this issue to collect some ideas to improve our performance.
Extending this idea, we could remove embedding vectors based on the impact that their removal has on the model performance. No matter what the access count is.
vq_layer
, and another 1x1 conv.vec_size
grows, the lower this number. Forvec_size
equal to the entire activation vector size, only one label can be assigned to every vector.