search

rosinusserrano / pml_vqvae

Repository for the course "Project Machine Learning" during WiSe 24/25 at TU Berlin consisting of a replication of the paper "Neural Discrete Representation Learning" (van den Oord et al., 2018).
0 stars 0 forks source link

Is there guidance/conditioning? #3

Closed rosinusserrano closed 1 week ago

rosinusserrano commented 3 weeks ago

In the paper they generated images for a given imagenet class, but in the model architecture they dont specify where you input the class id to the model huh

rosinusserrano commented 3 weeks ago

ChatGPT answer:

In VQ-VAE (Vector Quantized Variational Autoencoder), especially in early implementations like VQ-VAE-2, there's indeed no explicit mention of a guidance parameter or class-conditioning parameter for generating images in the style of a particular ImageNet class. Here's how it works:

  1. Two-Stage Training with a Hierarchical Structure: VQ-VAE-2 is trained in two stages: first, a VQ-VAE encoder-decoder is trained to reconstruct images. The encoder quantizes inputs by mapping them to discrete codes, capturing high-level features of the images. The decoder then reconstructs images based solely on these discrete codes. In the second stage, an autoregressive model (often PixelCNN) is trained to learn a distribution over these discrete codes. The combination of these models enables coherent image generation.
  2. Unconditional Generation: To generate images without direct conditioning, VQ-VAE-2's autoregressive prior is simply trained to learn the distribution of quantized codebook embeddings, essentially learning the "style" of the dataset (such as ImageNet) at each layer of its hierarchy. So, when generating images, the prior samples from the learned distribution over these codes and sends them to the decoder to produce an image.
  3. Indirect Class-Specific Sampling: The authors achieve class-specific generation by training separate prior models for each class, rather than conditioning a single model directly on class information. They split the ImageNet dataset into class subsets and train the priors on each class individually, rather than conditioning them explicitly. So, when sampling, they select a specific prior model corresponding to the target class, which allows for class-conditional image generation without additional conditioning inputs.
  4. No Guidance Parameter: VQ-VAE-2 doesn't use a guidance parameter in the way that modern conditional models (like CLIP-guided or classifier-guided diffusion models) do. Instead, it uses the autoregressive model trained on a specific subset (class) of data to bias the generation process towards that subset's characteristics, so no external "guidance" is needed.

In summary, they achieve the effect of conditioning by splitting the priors across classes rather than using explicit conditioning during generation. This workaround effectively allows VQ-VAE-2 to produce class-specific images without additional parameters in the decoder.

I don't think that we are implementing vqvae2 but still this makes sense to me. That the model is able to generate all kinds of images, but depending on how we choose our codes used for generation different kinds of images are generated (e.g. gray whale, ...) If we then train the prior to learn which groupings of codes will lead to the generation of what kind of images, the decoder output will generate those images.