SDXL-Lightning support

[francis2tm]

Hello, I wonder if stable-diffusion.cpp supports ByteDance/SDXL-Lightning?

[Mortezanavidi]

any news on this?

[mzwing]

I tested that, but got very bad result:

Command here: !./sd -m SDXL-Lightning/sdxl_lightning_4step.safetensors --vae SDXL-Lightning/sdxl_vae.safetensors -p "1girl, extremely beautiful, miniskirt, nsfw" -H 1024 -W 1024 --steps 4 -s -1 -v -o test.png

Update: Oops, I suddenly found that I had forgot to add --cfg-scale 1 parameter. It worked well now.

Command: ./sd -m SDXL-Lightning/sdxl_lightning_4step.safetensors --vae SDXL-Lightning/sdxl_vae.safetensors -p "1girl, extremely beautiful, miniskirt, nsfw" -H 1024 -W 1024 --cfg-scale 1 --steps 4 -s -1 -v -o test.png

Result here:

Tested on Google Colab (T4 GPU).

sdxl_lightning_2step.safetensors & sdxl_lightning_4step.safetensors are currently supported according to my test. But the quality of pictures maybe a little low.

[mzwing]

Seems like stable-diffusion.cpp does not support sdxl_lightning_1step.safetensors right now.

./sd -m SDXL-Lightning/sdxl_lightning_1step.safetensors --vae SDXL-Lightning/sdxl_vae.safetensors -p "1girl, extremely beautiful, miniskirt, nsfw" -H 1024 -W 1024 --cfg-scale 1 --steps 1 -s -1 -v -o test.png

Log:

Option: 
    n_threads:         2
    mode:              txt2img
    model_path:        SDXL-Lightning/sdxl_lightning_1step.safetensors
    wtype:             unspecified
    vae_path:          SDXL-Lightning/sdxl_vae.safetensors
    taesd_path:        
    esrgan_path:       
    controlnet_path:   
    embeddings_path:   
    output_path:       test.png
    init_img:          
    control_image:     
    controlnet cpu:    false
    strength(control): 0.90
    prompt:            1girl, extremely beautiful, miniskirt, nsfw
    negative_prompt:   
    min_cfg:           1.00
    cfg_scale:         1.00
    clip_skip:         -1
    width:             1024
    height:            1024
    sample_method:     euler_a
    schedule:          default
    sample_steps:      1
    strength(img2img): 0.75
    rng:               cuda
    seed:              1768825772
    batch_count:       1
    vae_tiling:        false
    upscale_repeats:   1
System Info: 
    BLAS = 1
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 0
    AVX512_VBMI = 0
    AVX512_VNNI = 0
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:122  - Using CUDA backend
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla T4, compute capability 7.5, VMM: yes
[INFO ] stable-diffusion.cpp:142  - loading model from 'SDXL-Lightning/sdxl_lightning_1step.safetensors'
[WARN ] model.cpp:682  - unknown format SDXL-Lightning/sdxl_lightning_1step.safetensors
[ERROR] stable-diffusion.cpp:148  - init model loader from file failed: 'SDXL-Lightning/sdxl_lightning_1step.safetensors'
new_sd_ctx_t failed

This may be because SDXL Lightning 1step version needs "sample" prediction type.

See: https://colab.research.google.com/github/camenduru/SDXL-Lightning-jupyter/blob/main/SDXL_Lightning_gradio_jupyter.ipynb

def generate_image(prompt, ckpt):

    checkpoint = checkpoints[ckpt][0]
    num_inference_steps = checkpoints[ckpt][1] 

    if num_inference_steps==1:
        # Ensure sampler uses "trailing" timesteps and "sample" prediction type for 1-step inference.
        pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing", prediction_type="sample")
    else:
        # Ensure sampler uses "trailing" timesteps.
        pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")

    pipe.unet.load_state_dict(torch.load(hf_hub_download(repo, checkpoint), map_location="cuda"))
    image = pipe(prompt, num_inference_steps=num_inference_steps, guidance_scale=0).images[0]
    return image

[mzwing]

You can get my quantized results here: https://huggingface.co/mzwing/SDXL-Lightning-GGUF

[mzwing]

Seems like stable-diffusion.cpp does not support sdxl_lightning_1step.safetensors right now.

Update: I suddenly realized that I had made a mistake: I did not download the 1step model at that time! That's why I got the error [ERROR] stable-diffusion.cpp:148 - init model loader from file failed: 'SDXL-Lightning/sdxl_lightning_1step.safetensors' new_sd_ctx_t failed

However, the model's generated image is very very bad. I thought that's because SDXL Lightning 1step version needs "sample" prediction type. See my comment above.

Generated by sdxl_lightning_1step_x0.q8_0.gguf.

You can still get the quantized version of the 1step model in my huggingface repo, but not recommended to use them.

[Green-Sky]

Yea the 1step needs custom code. But the other lightning variants work like a charm.

[Green-Sky]

Looks like for even better lightning generation we need a custom scheduler (for ref https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/15325 )

[teddybear082]

You can get my quantized results here: https://huggingface.co/mzwing/SDXL-Lightning-GGUF

hi do these ggufs need any vae or are they good to go alone? thanks for making them!

EDIT: nevermind worked fine without VAE for me! used q4_1.

[mzwing]

You can get my quantized results here: https://huggingface.co/mzwing/SDXL-Lightning-GGUF

hi do these ggufs need any vae or are they good to go alone? thanks for making them!

EDIT: nevermind worked fine without VAE for me! used q4_1.

Yes, when I quantised them I added the fp16 fix vae in it too :)

[teddybear082]

I'm so glad you made this because lightning is a perfect fit for the idea of quick lower requirement generation. I'm using it with koboldcpp since that has a chat and multimodal interface too. There, combining your 2 step model with LCM sampler looks really good to me.

[FSSRepo]

@mzwing I'll try to implement the missing scheduler, but I'm not exactly sure which of the models you've uploaded to Hugging Face I should try to see if I get the correct output.

sdxl_lightning_1step_x0.q8_0.gguf i will test.

[mzwing]

@mzwing ~I'll try to implement the missing scheduler, but I'm not exactly sure which of the models you've uploaded to Hugging Face I should try to see if I get the correct output.~

sdxl_lightning_1step_x0.q8_0.gguf i will test.

Maybe you also need to implement a prediction type if you use 1 step version to test (

The missing scheduler, I think, just uses for better quality of images generated by the SDXL-Lightning series.

See https://github.com/leejet/stable-diffusion.cpp/issues/198#issuecomment-2014357401

[Amin456789]

@mzwing could u please make some more 2step gguf q4.0 models? realistic unensored versions will be great also could u please make a q4.0 version gguf of original sdxl turbo? i tried to make a q5.1 version in the past and it didn't work for me, i'd like to test q4.0 version

[Amin456789]

hey guys, its my findings = sdxl turbo q5.0 works like a charm and it is 2.8gb only pony realism 4step lighting model with q4.0 is amazing too

[mzwing]

@mzwing could u please make some more 2step gguf q4.0 models? realistic unensored versions will be great

also could u please make a q4.0 version gguf of original sdxl turbo? i tried to make a q5.1 version in the past and it didn't work for me, i'd like to test q4.0 version

Sorry for my previous neglect. However now I have some other things to do and so may not be able to help you about that.

If you don't mind, I can help you quantize it a week later.

BTW could you explain more clearly about the "realistic unensored versions"? I have limited knowledge on them.

Edit: Seems that you have already quantized it? 👍

[Amin456789]

@mzwing could u please make some more 2step gguf q4.0 models? realistic unensored versions will be great also could u please make a q4.0 version gguf of original sdxl turbo? i tried to make a q5.1 version in the past and it didn't work for me, i'd like to test q4.0 version

Sorry about my previous neglect. However now I have some other things to do and so may not be able to help you about that.

If you don't mind, I can help you quantize it a week later.

BTW could you explain more clearly about the "realistic unensored versions"? I have limited knowledge on them.

Edit: Seems that you have already quantized it? 👍

greetings, thank u for ur answer, yes i managed to quantize turbo to q5.0 and it worked greatly. yes i got the models i needed, the uncensored ones i meant a model that is capable of nsfw stuff, but it is ok cause im using pony realism now which is great

[teddybear082]

@mzwing could u please make some more 2step gguf q4.0 models? realistic unensored versions will be great also could u please make a q4.0 version gguf of original sdxl turbo? i tried to make a q5.1 version in the past and it didn't work for me, i'd like to test q4.0 version

Sorry about my previous neglect. However now I have some other things to do and so may not be able to help you about that. If you don't mind, I can help you quantize it a week later. BTW could you explain more clearly about the "realistic unensored versions"? I have limited knowledge on them. Edit: Seems that you have already quantized it? 👍

greetings, thank u for ur answer, yes i managed to quantize turbo to q5.0 and it worked greatly. yes i got the models i needed, the uncensored ones i meant a model that is capable of nsfw stuff, but it is ok cause im using pony realism now which is great

Would you mind sharing links?

[Amin456789]

i like to do it but sadly my upload speed is in a bad shape, sorry