When performing VAE encoding on high-resolution images, there is a high probability that the original image will be changed and artifacts will be generated.

[leonary]

Expected Behavior

Since upscaling is an important feature, it is very important to remove these artifacts to preserve the original image content. These artifacts are difficult to remove at low denosing levels, making the result poor.

Actual Behavior

Original image https://files.catbox.moe/uxrle2.png After VAE encoding https://files.catbox.moe/4f9mbh.png Artifacts:

Steps to Reproduce

VAE_encode.json If you use any high-resolution image to encode with the SD15 series VAE, you should be able to observe this problem. The SDXL series VAE will not have this problem. There is a post on reddit discussing this issue.https://www.reddit.com/r/StableDiffusion/comments/1agd5pz/a_recent_post_went_viral_claiming_that_the_vae_is/ So this could be a problem with the VAE itself, but I'm curious if there are any tricks in the code that can avoid this artifact? Or is there a way to fix the VAE in the SD15 series?

Debug Logs

No debug logs

Other

No response

[leonary]

Found a repository that encodes with SDXL series VAE and decodes with SD15 series, but it has problems with artifacts and color shift. https://github.com/city96/SD-Latent-Interposer

[shawnington]

VAE encoding is not lossless.

[RandomGitUser321]

Since upscaling is an important feature, it is very important to remove these artifacts to preserve the original image content. These artifacts are difficult to remove at low denosing levels, making the result poor.

That is 100% to be expected. As Shawnington stated, vae encoding<>decoding is not lossless. When an image is vae encoded, it's in a very compressed state.

For example: A 1024x1024x24bit image would be exactly 3MB(before compression or lossless compression). Meanwhile, a vae encoded version of that image gets compressed to 128x128x4x16bit float precision in latent form(might be 32bit, but i think the 32bit modes are only for the calcuations, I'd have to double check). This means the latent space version of the image is 0.125 MB in size, which is a 24:1 reduction in size. When you then decode the image with a vae, you then take that image with 1/24th the information and reinflate it back to the original size.

So yeah, there will be losses and it won't get everything correct. This was one of the exciting things about the new 16 channel SD3 vae: encode<>decode quality.