search

Leminhbinh0209 / FinetuneVAE-SD

Fine-tune VAE of Stable Diffusion model
8 stars 1 forks source link

Latent regularization #1

Open Leminhbinh0209 opened 4 months ago

Leminhbinh0209 commented 4 months ago

TO-DO: As the denoiser of SD is not fine-tuned in this code, training the VAE alone on new dataset may shift the latent space. There may need to regularize the old the current latent vector with old latent vector in the teach-student: reg = (self.model.encode(x).saples() - old_vae.encode(x).sample().detach()).pow(2). But, I don't see its description in here, and at far as my experiment, there is no need for this.

eeyrw commented 2 months ago

Why not finetune the decoder only to keep latent intact?

Leminhbinh0209 commented 2 months ago

Oh right, that's a simple genius idea. Thank you for a great recommendation!

eeyrw commented 2 months ago

I thought there were some special reasons to use teacher-student way rather than simply freezing encoder part... By the way, freezing encoder part is common practice. https://huggingface.co/stabilityai/sd-vae-ft-mse-original said: ...To keep compatibility with existing models, only the decoder part was finetuned; the checkpoints can be used as a drop-in replacement for the existing autoencoder..