[Error]Fail to run

[Micraow]

I followed the guidance and downloaded the SDXL Turbo OpenVINO int8 and TAESDXL OpenVINO for offline use. However, I encountered with an error. Here are my outputs:

Starting FastSD CPU please wait...
Found python command
Python version : 3.11.6
Running on Linux platform
OS: Linux-6.6.7-zen1-1-zen-x86_64-with-glibc2.38
Processor: 
Using device : cpu
Found 5 LCM models in config/lcm-models.txt
Found 7 stable diffusion models in config/stable-diffusion-models.txt
Found 3 LCM-LoRA models in config/lcm-lora-models.txt
Found 5 OpenVINO LCM models in config/openvino-lcm-models.txt
Torch datatype : torch.float32
Starting web UI mode
Error:/mnt/btrfs/Fastsdcpu/sdxl-turbo-openvino-int8/ Model not found in configuration file,so using first model : stabilityai/sd-turbo
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Reshape and compile
{'diffusion_task': 'text_to_image',
 'guidance_scale': 1.0,
 'image_height': 1024,
 'image_width': 1024,
 'inference_steps': 3,
 'init_image': None,
 'lcm_lora': {'base_model_id': 'Lykon/dreamshaper-8',
              'lcm_lora_id': 'latent-consistency/lcm-lora-sdv1-5'},
 'lcm_model_id': '/mnt/btrfs/Fastsdcpu/sdxl-turbo-openvino-int8/',
 'negative_prompt': '',
 'number_of_images': 1,
 'openvino_lcm_model_id': '/mnt/btrfs/Fastsdcpu/sdxl-turbo-openvino-int8/',
 'prompt': 'a cat',
 'seed': -1,
 'strength': 0.6,
 'use_lcm_lora': False,
 'use_offline_model': True,
 'use_openvino': True,
 'use_safety_checker': False,
 'use_seed': False,
 'use_tiny_auto_encoder': True}
***** Init Text to image (OpenVINO) - /mnt/btrfs/Fastsdcpu/sdxl-turbo-openvino-int8/ *****
Compiling the vae_decoder to CPU ...
Compiling the unet to CPU ...
Compiling the text_encoder_2 to CPU ...
Compiling the vae_encoder to CPU ...
Compiling the text_encoder to CPU ...
Using Tiny Auto Encoder (OpenVINO)
Pipeline : OVStableDiffusionPipeline {
  "_class_name": "OVStableDiffusionPipeline",
  "_diffusers_version": "0.23.0",
  "_name_or_path": "stabilityai/sdxl-turbo",
  "force_zeros_for_empty_prompt": true,
  "requires_aesthetics_score": false,
  "scheduler": [
    "diffusers",
    "EulerAncestralDiscreteScheduler"
  ],
  "text_encoder": [
    "optimum",
    "OVModelTextEncoder"
  ],
  "text_encoder_2": [
    "optimum",
    "OVModelTextEncoder"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "tokenizer_2": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "optimum",
    "OVModelUnet"
  ],
  "vae_decoder": [
    "optimum",
    "OVModelVaeDecoder"
  ],
  "vae_encoder": [
    "optimum",
    "OVModelVaeEncoder"
  ]
}

Using OpenVINO
/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/optimum/intel/openvino/modeling_diffusion.py:565: FutureWarning: `shared_memory` is deprecated and will be removed in 2024.0. Value of `shared_memory` is going to override `share_inputs` value. Please use only `share_inputs` explicitly.
  outputs = self.request(inputs, shared_memory=True)
  0%|                                                                                                                                             | 0/3 [00:00<?, ?it/s]/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/optimum/intel/openvino/modeling_diffusion.py:599: FutureWarning: `shared_memory` is deprecated and will be removed in 2024.0. Value of `shared_memory` is going to override `share_inputs` value. Please use only `share_inputs` explicitly.
  outputs = self.request(inputs, shared_memory=True)
  0%|                                                                                                                                             | 0/3 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/gradio/routes.py", line 442, in run_predict
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/gradio/blocks.py", line 1392, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/gradio/blocks.py", line 1097, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/gradio/utils.py", line 703, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/src/frontend/webui/text_to_image_ui.py", line 53, in generate_text_to_image
    images = future.result()
             ^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/src/context.py", line 36, in generate_text_to_image
    images = self.lcm_text_to_image.generate(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/src/backend/lcm_text_to_image.py", line 302, in generate
    result_images = self.pipeline(
                    ^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/optimum/intel/openvino/modeling_diffusion.py", line 687, in __call__
    return StableDiffusionPipelineMixin.__call__(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/optimum/pipelines/diffusers/pipeline_stable_diffusion.py", line 357, in __call__
    noise_pred = self.unet(sample=latent_model_input, timestep=timestep, encoder_hidden_states=prompt_embeds)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/optimum/intel/openvino/modeling_diffusion.py", line 599, in __call__
    outputs = self.request(inputs, shared_memory=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/openvino/runtime/ie_api.py", line 384, in __call__
    return self._infer_request.infer(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/micraow/fastsdcpu/env/lib/python3.11/site-packages/openvino/runtime/ie_api.py", line 143, in infer
    return OVDict(super().infer(_data_dispatch(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src/inference/src/infer_request.cpp:82:
Exception from src/inference/src/infer_request.cpp:74:
[ GENERAL_ERROR ] Can't set input blob with name: encoder_hidden_states, because model input (shape=[?,77,2048]) and blob (shape=(1.77.768)) are incompatible

At first, I guessed that has some thing to do with image size, so I changed the image size from 512512 to 10241024. It didn't work.

By the way, I found that this program doesn't use my downloaded tae model, so I changed the path in src/constants.py , then I found that even though I am using SDXL turbo model, the program uses the taesd instead of taesdxl. So I changed the taesd_dir in src/backend/openvino/pipelines.py to make it uses TAESDXL OpenVINO

Does anyone knows how to fix it? Thank you!

p.s. This is quite a great project though it hasn't got many stars. I think maybe I can introduce this project on my blog: Peng's Blog as well as an instruction in Chinese.

UPDATE: I disabled the TAE, but the problem exist

[rupeshs]

@Micraow Thank you for using Fast SD CPU, can you try a fresh install?

[Micraow]

OK, tried again. This time I picked the given model in the menu.It tried to download, but failed. (You know, in China, we have the GFW, which prohibits us from accessing github and huggingface.That's why I have to use offline). I put the previous downloaded model into huggingface cache directory. It worked.

Then I tried to use offline model as the Readme says, but it still failed, and the program handle the SDXL model as SD model

[rupeshs]

OK, I hope you are using the latest release https://github.com/rupeshs/fastsdcpu/releases/tag/v1.0.0-beta.23. Please try this, delete the config/settings.yaml file start the webui, and ensure that the following settings are ok.

[Micraow]

Well, work now!Thank you! By the way, I find the SDXL Trubo on my computer is much slower than yours. My computer has an 10th Gen Intel i3 and 8GB RAM and 25GB swap on SSD, and it took 1782s to generate a robot of 512*512. (without TAESD)

[rupeshs]

@Micrasow First time it will take more time because of model loading and compiling after that you will get good speed.

[Micraow]

Oops, I find openvino still compile the unet and encoders again,even though I have set cache dir...

在 2023-12-24 22:07:58，"Rupesh Sreeraman" @.***> 写道：

@Micrasow First time it will take more time because of model loading and compiling after that you will get good speed.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>

[rupeshs]

It will compile at startup and when there is size change or model change etc