Bad quality with RealVisXL_V4.0_Lightning

[Swarzox]

Describe the bug

Using _text_to_image_sdxllight.py everything work fine with ByteDance/SDXL-Lightning but when I try to make it works with SG161222/RealVisXL_V4.0_Lightning I get very poor quality

Your environment

Ubuntu 20.04, cuda 11.8, python 3.10, rtx 4090

OneFlow version info

libibverbs not available, ibv_fork_init skippedpath: ['/usr/local/lib/python3.10/dist-packages/oneflow']version: 0.9.1.dev20240515+cu118 git_commit: ec7b682 cmake_build_type: Release rdma: True mlir: True enterprise: True

How To Reproduce

import os
import argparse
import time

import torch
from safetensors.torch import load_file
from diffusers import StableDiffusionXLPipeline, DPMSolverSinglestepScheduler
from onediffx import compile_pipe, save_pipe, load_pipe
from huggingface_hub import hf_hub_download

try:
    USE_PEFT_BACKEND = diffusers.utils.USE_PEFT_BACKEND
except Exception as e:
    USE_PEFT_BACKEND = False

OUTPUT_TYPE = "pil"
n_steps = 4

from diffusers import UNet2DConditionModel
unet = UNet2DConditionModel.from_config("stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet").to("cuda", torch.float16)
unet.load_state_dict(load_file("diffusion_pytorch_model.safetensors", device="cuda"))
pipe = StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.scheduler = DPMSolverSinglestepScheduler.from_config(pipe.scheduler.config, use_karras_sigmas=True)

if pipe.vae.dtype == torch.float16 and pipe.vae.config.force_upcast:
    pipe.upcast_vae()

# Compile the pipeline
pipe = compile_pipe(pipe,)

print("Warmup with running graphs...")
torch.manual_seed(1)
image = pipe(
    prompt="A girl smiling",
    height=1024,
    width=1024,
    num_inference_steps=n_steps,
    guidance_scale=0,
    output_type=OUTPUT_TYPE,
).images

while True:
    # Prompt the user for input
    user_prompt = input("Enter a new prompt (or type 'exit' to quit): ")
    if user_prompt.lower() == 'exit':
        break

    # Normal run with new prompt
    print("Generating image...")
    torch.manual_seed(1)
    start_t = time.time()
    image = pipe(
        prompt=user_prompt,
        height=1024,
        width=1024,
        num_inference_steps=n_steps,
        guidance_scale=1,
        output_type=OUTPUT_TYPE,
    ).images

    end_t = time.time()
    print(f"e2e ({n_steps} steps) elapsed: {end_t - start_t} s")

    # Save the image
    image[0].save("out.png")
    print(f"Image saved as out.png")

    if args.save_graph:
        print("Saving graphs...")
        save_pipe(pipe, args.save_graph_dir)

Additional context

I tried to change the scheduler to DPM++ SDE Karras with DPMSolverSinglestepScheduler

I chose the safetensors diffusion_pytorch_model.safetensors available here : https://huggingface.co/SG161222/RealVisXL_V4.0_Lightning/tree/main/unet

[strint]

enterprise: True

You are using the int8 (quantized) version of mode?

@Swarzox Can you post some image here? We haven't run with SG161222/RealVisXL_V4.0_Lightning before.

[doombeaker]

I ran RealVis XL 4.0 with OneDiff quant, the outputs quality is without any problem.

I recommend that to try the ComfyUI first

[strint]

No new feedback, please feel free to reopen it.