search

junhsss / consistency-models

A Toolkit for OpenAI's Consistency Models.
https://arxiv.org/abs/2303.01469
MIT License
195 stars 11 forks source link

question regarding the training #9

Closed yuanzhi-zhu closed 1 year ago

yuanzhi-zhu commented 1 year ago

Thanks for your excellent work! I have several questions regarding the training process.

  1. I noticed that in your wandb, the samples from steps=131 are of quite good quality, how is it so?
  2. can you provide the training config for the experiments in your wandb, eg. the 'cifar10'?
  3. I found that when setting the loss function to mse, the generated samples are kind of noisy, is this inevitable?

massive thanks in advance

junhsss commented 1 year ago

Hi @yuanzhi-zhu.

  1. It's actually a bug in wandb 😅. If there are no logged samples at step 131, it displays the final samples of that experiment instead.
  2. You can find the training arguments in here and the model configuration in the model repository. The weird batch size was chosen to maximize GPU memory usage.
  3. $L_2$ loss tends to make things blurry because it aims to learn the average in the pixel space. That's why some people decided to train the discriminator for this kind of problem! I wouldn't say that it's inevitable, but structural bias of NNs could definitely make it less noisy. After all, LPIPS loss is just $L_2$ loss in the latent space.

Please don't hesitate to reach out if you have any further questions!

yuanzhi-zhu commented 1 year ago

Hi @junhsss,

Thank you so much for your clarification! (I am new to wandb