[Feature Request]: openvino_accelerate.py script need support released torch2.1.0

[taotaow]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

current torch version is 2.1.0.dev20230816+cpu torch 2.1.0 had released,can you support it? when upgrade to 2.1.0 version,text2img can't run,it assert error:

Traceback (most recent call last): File "D:\stable-diffusion-webui\modules\call_queue.py", line 57, in f res = list(func(*args, kwargs)) File "D:\stable-diffusion-webui\modules\call_queue.py", line 36, in f res = func(*args, *kwargs) File "D:\stable-diffusion-webui\modules\txt2img.py", line 52, in txt2img processed = modules.scripts.scripts_txt2img.run(p, args) File "D:\stable-diffusion-webui\modules\scripts.py", line 601, in run processed = script.run(p, script_args) File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 1134, in run processed = process_images_openvino(p, model_config, vae_ckpt, p.sampler_name, enable_caching, openvino_device, mode, is_xl_ckpt, refiner_ckpt, refiner_frac) File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 884, in process_images_openvino output = shared.sd_diffusers_model( File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\diffusers\pipelines\stable_diffusion\pipeline_stable_diffusion.py", line 680, in call noise_pred = self.unet( File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch_dynamo\eval_frame.py", line 328, in _fn return fn(args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\diffusers\models\unet_2d_condition.py", line 734, in forward def forward( File "D:\stable-diffusion-webui\venv\lib\site-packages\torch_dynamo\eval_frame.py", line 328, in _fn return fn(args, kwargs) File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 188, in _call res = execute(compiled_model, args, executor="openvino", File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 217, in execute return openvino_execute_partitioned(gm, args, executor_parameters=executor_parameters, file_name=file_name) File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 319, in openvino_execute_partitioned return partitioned_modulessignature File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\fx\graph_module.py", line 678, in call_wrapped return self._wrapped_call(self, *args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\fx\graph_module.py", line 284, in call raise e File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\fx\graph_module.py", line 274, in call return super(self.cls, obj).call(*args, *kwargs) # type: ignore[misc] File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl return self._call_impl(args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File ".35", line 23, in forward fused_30 = self.fused_30(arg2_1, getitem_12, getitem_800); getitem_12 = getitem_800 = None File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 240, in call result = openvino_execute(self.gm, args, executor_parameters=self.executor_parameters, partition_id=self.partition_id, file_name=self.file_name) File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 284, in openvino_execute compiled = openvino_compile(gm, *args, model_hash_str=model_hash_str, file_name=file_name) File "D:\stable-diffusion-webui\scripts\openvino_accelerate.py", line 443, in openvino_compile om.inputs[idx].get_node().set_element_type(dtype_mapping[input_data.dtype]) IndexError: list index out of range

Proposed workflow

1. Go to ....
2. Press ....
3. ...

Additional information

No response

[qiacheng]

Hi @taotaow, yes we are currently working on getting an openvino nightly build released with the scaled_dot_product_flash_attention operator support to support torch 2.1.0

[qiacheng]

@taotaow

temporary workaround

pip install torch==2.1.0 torchvision==0.16.0 https://github.com/cavusmustafa/openvino/raw/custom_sdpa_optimizations/openvino-2023.2.0-12829-cp310-cp310-win_amd64.whl

Note that this wheel package only works on windows with python 3.10 .. for Linux we need to wait for the fixes to be merged onto openvino master branch then we can do pip install openvino-nightly later.

[qiacheng]

Torch 2.1.0 is now supported with openvino-nightly==2023.2.0.dev20231101 or newer

[taotaow]

thank you,It worked