search

leejet / stable-diffusion.cpp

Stable Diffusion and Flux in pure C/C++
MIT License
3.31k stars 279 forks source link

[Feature Request] Support for SDXS-512, allowing for real-time image generation on the CPU (~0.6 seconds per image) #260

Open JohnAlcatraz opened 4 months ago

JohnAlcatraz commented 4 months ago

It would be great if support for the SDXS-512 model could be added: https://github.com/IDKiro/sdxs

Especially for quickly generating images on the CPU, this is a major new development, so it would fit this project really well. https://github.com/rupeshs/fastsdcpu already supports it (requires Python though and really isn't lightweight and dependency-free like stable-diffusion.cpp).

I benchmarked SDXS-512 in FastSDCPU on my CPU (3950X):

FastSDCPU Regular: RAM Usage: 700 MB Image Generation Time (Latency): 2 seconds

FastSDCPU OpenVINO: RAM Usage: 3.6 GB Image Generation Time (Latency): 0.6 seconds

So it's really very fast, and even super memory efficient (when not using OpenVINO). And it even looks much better than the best reasonably fast results I can get with stablediffusion.cpp, which is using Dreamshaper LCM with ~5 steps, 2 GB RAM usage and 19 seconds generation time. SDXS-512 looks better, is 20 times faster and uses only 1/3 the RAM - and that's with a Python implementation.

Amin456789 commented 3 months ago

indeed, if its gonna faster than sdxl turbo 512x512 with 1step then it is gonna be great to have it

have u tried it already? maybe its gonna work, we got sdxl lighitng and pony realsim to work without sd cpp get a new update. maybe this gonna work too

Amin456789 commented 3 months ago

@idkiro could u please take a look at this, we can use this for koboldcpp for chat and image generation, thank u

IDKiro commented 3 months ago

@IDKiro could u please take a look at this, we can use this for koboldcpp for chat and image generation, thank u

Yes, SDXS is much faster than SD Turbo, and I recommend using the SDXS-512-DreamShaper we put out, which is compatible with original SD1.5 VAE and Tiny VAE. The SDXS-512-DreamShaper is based on SD 1.5, but with changes in the structure of the UNet, and we have marked the changes in the diffusers configuration file: { "block_out_channels": [ 320, 640, 1280 ], "down_block_types": [ "DownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D" ], "mid_block_type": null, "layers_per_block": 1, "up_block_types": [ "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "UpBlock2D" ], }

Amin456789 commented 3 months ago

thank u for ur answer idkiro, hope someone add this to sd cpp @leejet can u please make changes to support this one as well, thank u