SDXL support

[solitaryTian]

Is this method support SDXL? I just replace the model with sdxl base. And it raise error

Traceback (most recent call last): File "/dfs/comicai/songtao.tian/StreamDiffusion/streamdiffusion_test.py", line 42, in stream(init_image) File "/root/miniconda3/envs/streamdiffusion/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "/dfs/comicai/songtao.tian/StreamDiffusion/src/streamdiffusion/pipeline.py", line 461, in call x_0_pred_out = self.predict_x0_batch(x_t_latent) File "/dfs/comicai/songtao.tian/StreamDiffusion/src/streamdiffusion/pipeline.py", line 399, in predict_x0_batch x_0_pred_batch, model_pred = self.unet_step(x_t_latent, t_list) File "/dfs/comicai/songtao.tian/StreamDiffusion/src/streamdiffusion/pipeline.py", line 313, in unet_step model_pred = self.unet( File "/root/miniconda3/envs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, *kwargs) File "/root/miniconda3/envs/streamdiffusion/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(args, kwargs) File "/root/miniconda3/envs/streamdiffusion/lib/python3.10/site-packages/diffusers/models/unet_2d_condition.py", line 967, in forward if "text_embeds" not in added_cond_kwargs: TypeError: argument of type 'NoneType' is not iterable

[teftef6220]

Thanks for the comment.

We do not currently support SD_XL.

The development team will look into supporting SD_XL.

[jayavanth]

@teftef6220 can you tell us what changes are needed? Maybe we can help

[teftef6220]

Thanks for the comments, To achieve higher speeds, we need to develop an SDXL-compatible TensorRT and model architecture must be modified. This will require significant changes to the pipeline, which could take a long time.

[bobby20180331]

I am working on Have stream support sdxl-1.0-base model. However, error accured. any suggestions? : ../diffusers/loaders/unet.py", line 260, in load_attn_procs attn_processor = getattr(attn_processor, sub_key) File "../site-packages/torch/nn/modules/module.py", line 1695, in getattr raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'") AttributeError: 'DownBlock2D' object has no attribute 'attentions'

[hkn-g]

I have added SDXL support to the pipeline and wrapper based on @ApolloRay 's code (thanks for that!) (https://github.com/cumulo-autumn/StreamDiffusion/issues/114#issuecomment-1953697528). I have also added single image and real-time image generation examples for sdxl-turbo. Check out my sdxl branch.

https://github.com/hkn-g/StreamDiffusion/tree/sdxl

sdxl-turbo img2img looks fine, but there are some other issues, such as TensorRT not working, and text2img giving an error with an SDXL model. However, in img2img mode, it works without an input image.

[tebjan]

@hkn-g I got the same error as @solitaryTian. Do you happen to know what the issue could be? I am on the sdxl branch Windows 11 with Cuda 12.1 and I have created the venv as described in the readme of this repo: https://github.com/hkn-g/StreamDiffusion/tree/sdxl

Could this be of help? https://github.com/huggingface/diffusers/issues/4649#issuecomment-1890761454

[hkn-g]

@tebjan With Cuda 12.1 I couldn't get a proper working setup, I got different errors each time. Then I installed Cuda 11.8 to get a fully working environment, that might be the problem. If only the SDXL model in txt2img mode gives this error, I haven't looked at that part in detail yet.

[tebjan]

With Cuda 12.1 I couldn't get a proper working setup, I got different errors each time.

What errors did you get? With the fix that I PRed to your branch, your example for img2img works on my machine with win11, cuda 12.1, and a venv on a laptop with RTX 4090 gpu.

@solitaryTian a fix for the original issue is here: https://github.com/hkn-g/StreamDiffusion/pull/1

I am just wondering why sdxl-turbo works for others that didn't apply this fix... If anyone knows, please let me know.

[hkn-g]

What errors did you get?

I don't remember exactly, but I usually got dependency related errors related to xformers or TensorRT, but I think that was one of the errors somewhere. :)

By the way your commit (https://github.com/tebjan/StreamDiffusion/commit/a649ca55ea517d007e1f832a21b2830bf18da219) also fixed the txt2img issue in sdxl-turbo, thanks. I also updated the txt2img example for sdxl.

[chuck-ma]

It seems that sdxl (such as RunDiffusion/Juggernaut-XL-v9) doesn't work, but sdxl turbo works well.

[mflux]

Hi @hkn-g I am using your tree + the above patch but I still could not get examples\screen\main.py to work with sdxl-turbo.

Running the command:

screen\main.py --enable_similar_image_filter=False --model_id_or_path=stabilityai/sdxl-turbo

Gives me the following error:

Traceback (most recent call last):
  File "C:\Users\Flux\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Users\Flux\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Flux\strd-sdxl\examples\screen\main.py", line 194, in image_generation_process
    output_images = stream.stream(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\streamdiffusion\pipeline.py", line 501, in __call__
    x_0_pred_out = self.predict_x0_batch(x_t_latent)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\streamdiffusion\pipeline.py", line 438, in predict_x0_batch
    x_0_pred_batch, model_pred = self.unet_step(x_t_latent, t_list, added_cond_kwargs=added_cond_kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\streamdiffusion\pipeline.py", line 329, in unet_step
    model_pred = self.unet(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\diffusers\models\unet_2d_condition.py", line 1075, in forward
    sample, res_samples = downsample_block(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\diffusers\models\unet_2d_blocks.py", line 1160, in forward
    hidden_states = attn(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\diffusers\models\transformer_2d.py", line 392, in forward
    hidden_states = block(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\diffusers\models\attention.py", line 323, in forward
    attn_output = self.attn2(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\diffusers\models\attention_processor.py", line 522, in forward
    return self.processor(
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\diffusers\models\attention_processor.py", line 1137, in __call__
    key = attn.to_k(encoder_hidden_states, *args)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Flux\strd-sdxl\streamdiffusion\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (154x768 and 2048x640)

[mflux]

I got it working by replacing wrapper.py StableDiffusionPipeline references with StableDiffusionXLPipeline. I reckon that ought to be an option to be passed to wrapper.

But the image looks messed up and the framerate is not good (5fps compared to 22fps I get wtih SDTurbo). I'm not 100% sure if I'm doing it right.

[hkn-g]

Hi @mflux Are you sure it works on the sdxl branch ? Because wrapper.py already automatically decides the pipeline (StableDiffusionXLPipeline or not) by checking the model name, or you can manually pass the sdxl=True parameter to the wrapper.