Compatibility of TAESD VAE Encoder Output with SD Original VAE

madebyollin / taesd

Tiny AutoEncoder for Stable Diffusion

MIT License

579 stars 27 forks source link

Compatibility of TAESD VAE Encoder Output with SD Original VAE #25

Open aidenyzhang opened 3 months ago

aidenyzhang commented 3 months ago

Hello,

Thank you for your work on TAE-SD. I have a quick question about compatibility:

Is the VAE encoder output from TAE-SD the same as that from SD-VAE-FT-MSE? I have a U-Net trained on SD-VAE-FT-MSE, and I am wondering if it is necessary to retrain the U-Net to work with TAE-SD.

Any clarification on this would be greatly appreciated.

Thank you!

madebyollin commented 3 months ago

Yeah, the encoder outputs should be compatible (though not 100% identical), so any model that was trained with SD-VAE can also be used with TAESD. I posted an example here showing that you can even encode an image with TAESD and decode with the SD-VAE.