Is there a way to use custom VAE?

apple / ml-stable-diffusion

Stable Diffusion with Core ML on Apple Silicon

MIT License

16.79k stars 935 forks source link

Is there a way to use custom VAE? #79

Open upstroke-version opened 1 year ago

upstroke-version commented 1 year ago

Lots of model use their custom fine tuned VAE to get color and texture right, for example (https://huggingface.co/Linaqruf/anything-v3.0/blob/main/Anything-V3.0.vae.pt). Is there a way to use these fine tuned VAE on apple's ml-stable-diffusion? Can torch2coreml script do this job? I've used other's pre-converted coreml model and most of them looks like they are missing proper VAE.

icpryde commented 1 year ago

Ya this is very important really need this support

rovo79 commented 1 year ago

It seems like this feature request is now possible via the --custom-vae-version flag. https://github.com/apple/ml-stable-diffusion#model-conversion

Ex. --custom-vae-version madebyollin/sdxl-vae-fp16-fix