search

nod-ai / SHARK

SHARK - High Performance Machine Learning Distribution
Apache License 2.0
1.41k stars 168 forks source link

UNKNOWN; VkResult=4294967283 #1434

Open Cangjin-Zhao opened 1 year ago

Cangjin-Zhao commented 1 year ago

Hello, when i try to run shark for my 7900xt, but the lastest stabel version has the same issue shark_tank local cache is located at C:\Users\Cangjin.local/shark_tank/ . You may change this by setting the --local_tank_cache= flag vulkan devices are available. cuda devices are not available. diffusers\models\cross_attention.py:30: FutureWarning: Importing from cross_attention is deprecated. Please import from diffusers.models.attention_processor instead. Running on local URL: http://0.0.0.0:8080

To create a public link, set share=True in launch(). Found device AMD Radeon HD 7900 Series. Using target triple rdna2-unknown-windows. Using tuned models for Linaqruf/anything-v3.0/fp16/vulkan://00000000-1300-0000-0000-000000000000. loading existing vmfb from: F:\SHARK\euler_scale_model_input_1_512_512_vulkan_fp16.vmfb WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_api_dump.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_gfxreconstruct.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_synchronization2.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_validation.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_screenshot.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_profiles.json invalid layer manifest file version 1.2.1. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_shader_object.json invalid layer manifest file version 1.2.0. May cause errors. loading existing vmfb from: F:\SHARK\euler_step_1_512_512_vulkan_fp16.vmfb WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_api_dump.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_gfxreconstruct.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_synchronization2.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_validation.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_screenshot.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_profiles.json invalid layer manifest file version 1.2.1. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_shader_object.json invalid layer manifest file version 1.2.0. May cause errors. use_tuned? sharkify: True _1_77_512_512_fp16_tuned_anything-v3 WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_api_dump.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_gfxreconstruct.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_synchronization2.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_validation.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_screenshot.json invalid layer manifest file version 1.2.0. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_profiles.json invalid layer manifest file version 1.2.1. May cause errors. WARNING: [Loader Message] Code 0 : loaderAddLayerProperties: C:\VulkanSDK\1.3.246.1\Bin\VkLayer_khronos_shader_object.json invalid layer manifest file version 1.2.0. May cause errors. Traceback (most recent call last): File "gradio\routes.py", line 401, in run_predict File "gradio\blocks.py", line 1302, in process_api File "gradio\blocks.py", line 1039, in call_function File "anyio\to_thread.py", line 31, in run_sync File "anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread File "anyio_backends_asyncio.py", line 867, in run File "gradio\utils.py", line 491, in async_iteration File "ui\txt2img_ui.py", line 177, in txt2img_inf File "apps\stable_diffusion\src\pipelines\pipeline_shark_stable_diffusion_txt2img.py", line 114, in generate_images File "apps\stable_diffusion\src\pipelines\pipeline_shark_stable_diffusion_utils.py", line 376, in encode_prompts_weight File "apps\stable_diffusion\src\pipelines\pipeline_shark_stable_diffusion_utils.py", line 86, in load_clip File "apps\stable_diffusion\src\models\model_wrappers.py", line 601, in clip File "apps\stable_diffusion\src\models\model_wrappers.py", line 594, in clip File "apps\stable_diffusion\src\models\model_wrappers.py", line 531, in get_clip File "apps\stable_diffusion\src\utils\utils.py", line 120, in compile_through_fx File "apps\stable_diffusion\src\utils\utils.py", line 47, in _load_vmfb File "shark\shark_inference.py", line 207, in load_module File "shark\iree_utils\compile_utils.py", line 333, in load_flatbuffer File "shark\iree_utils\compile_utils.py", line 309, in get_iree_module File "iree\runtime\system_api.py", line 255, in add_vm_module File "iree\runtime\system_api.py", line 252, in add_vm_modules SystemExit: Error registering modules: D:\a\SHARK-Runtime\SHARK-Runtime\c\runtime\src\iree\hal\drivers\vulkan\native_executable.cc:157: UNKNOWN; VkResult=4294967283; while invoking native function hal.executable.create; while calling import; [ 1] native hal.executable.create:0 - [ 0] bytecode module@1:1072 -

monorimet commented 1 year ago

I haven't seen these vulkan loader messages before but it looks like they may be relevant to the failure.

Runtime\c\runtime\src\iree\hal\drivers\vulkan\native_executable.cc:157: UNKNOWN; VkResult=4294967283; while invoking native function hal.executable.create; while calling import; usually corresponds to resource exhaustion.

Can you help us reproduce this error? I see that this is on anythingv3 which should be fine especially on a 7900. What image dimensions are you generating? Any other context would help :)

AlexanderMusienko commented 1 year ago

Same issue, equal logs. (AMD r9 280x, adrenalin 22.6.1 drivers). Reproduction: generating any to image with any model