search

nod-ai / SHARK

SHARK - High Performance Machine Learning Distribution
Apache License 2.0
1.41k stars 168 forks source link

release 693 - on RX 6700 XT - select api version mismatch - tuned still has issues - tries to recompile over and over again #1327

Open GoZippy opened 1 year ago

GoZippy commented 1 year ago 
Found device AMD Radeon RX 6700 XT. Using target triple rdna2-unknown-windows
Using tuned models for CompVis/stable-diffusion-v1-4/fp16/vulkan://00000000-0300-0000-0000-000000000000.
No vmfb found. Compiling and saving to E:\ImageAI\shark\euler_scale_model_input_1_512_512fp16.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in E:\ImageAI\shark\euler_scale_model_input_1_512_512fp16.vmfb.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
No vmfb found. Compiling and saving to E:\ImageAI\shark\euler_step_1_512_512fp16.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in E:\ImageAI\shark\euler_step_1_512_512fp16.vmfb.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
use_tuned? sharkify: True
_1_64_512_512_fp16_tuned_duchaitenwlopFantasy_v10Fp32NoEma
Checkpoint already loaded at :  E:/ImageAI/shark/models/duchaitenEma
No vmfb found. Compiling and saving to E:\ImageAI\shark\clip_1_64_512_512_fp16_tuned_duchaitNoEma_vulkan.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in E:\ImageAI\shark\clip_1_64_512_512_fp16_tuned_duchaitNoEma_vulkan.vmfb.

WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
Loading Winograd config file from  C:\Users\me\.local/shark_tank/configs\unet_winograd_vulkan.json
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:00<00:00, 1.86kB/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 107/107 [00:00<00:00, 7.36kB/s]
Loading lowering config file from  C:\Users\me\.local/shark_tank/configs\unet_v1_4_fp16_vulkan_rdna2.json
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24.6k/24.6k [00:00<00:00, 303kB/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24.6k/24.6k [00:00<00:00, 456kB/s]
Applying tuned configs on unet_1_64_512_512_fp16_tuned_

then it gets stuck and tries to compile again and finally it seems to work...

Loading lowering config file from  C:\Users\gotad\.local/shark_tank/configs\unet_v1_4_fp16_vulkan_rdna2.json
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24.6k/24.6k [00:00<00:00, 303kB/s]
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24.6k/24.6k [00:00<00:00, 456kB/s]
Applying tuned configs on unet_1_64_512_512_fp16_tuned_duchaiNoEma_vulkan
No vmfb found. Compiling and saving to E:\ImageAI\shark\unet_1_64_512_512_fp16_tuned_duchaiNoEma_vulkan.vmfb
Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args
Saved vmfb in E:\ImageAI\shark\unet_1_64_512_512_fp16_tuned_duchaiNoEma_vulkan.vmfb.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
WARNING: [Loader Message] Code 0 : Layer VK_LAYER_OBS_HOOK uses API version 1.2 which is older than the application specified API version of 1.3. May cause issues.
27it [00:08,  3.02it/s]
torch\fx\node.py:251: UserWarning: Trying to prepend a node to itself. This behavior has no effect on the graph.
  warnings.warn("Trying to prepend a node to itself. This behavior has no effect on the graph.")
monorimet commented 1 year ago

Yeah, I also see this on 693. Working on a fix, need to be careful about how we do first-time tuned compilation. Thanks for posting.

monorimet commented 1 year ago

Might not be a way around doing some retries to find the right model configuration. Do you have to restart the program to get a successful run or does it just take a while the first time? I'm not too sure about the VK loader messages, but seems like a driver issue?

(if you see Applying tuned configs on unet_1_64_512_512_fp16_tuned_duchaiNoEma_vulkan, it may take a bit to apply the configs. up to a minute or two, but not longer than that.)

SocialSophia commented 1 year ago

Might not be a way around doing some retries to find the right model configuration. Do you have to restart the program to get a successful run or does it just take a while the first time? I'm not too sure about the VK loader messages, but seems like a driver issue?

(if you see Applying tuned configs on unet_1_64_512_512_fp16_tuned_duchaiNoEma_vulkan, it may take a bit to apply the configs. up to a minute or two, but not longer than that.)

Yeah I have the same issue honestly, this was my code: To create a public link, set share=True in launch(). Found device AMD Radeon RX 6700 XT. Using target triple rdna2-unknown-windows. Using tuned models for CompVis/stable-diffusion-v1-4/fp16/vulkan://00000000-0300-0000-0000-000000000000. Downloading (…)cheduler_config.json: 100%|████████████████████████████████████████████████████| 313/313 [00:00<?, ?B/s] huggingface_hub\file_download.py:133: UserWarning: huggingface_hub cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\Acqua.cache\huggingface\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the HF_HUB_DISABLE_SYMLINKS_WARNING environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations. To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to see activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development loading existing vmfb from: C:\SHARK Stable Diffusion\SHARK\euler_scale_model_input_1_512_512_vulkan_fp16.vmfb WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files. loading existing vmfb from: C:\SHARK Stable Diffusion\SHARK\euler_step_1_512_512_vulkan_fp16.vmfb WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files. use_tuned? sharkify: True _1_64_512_512_fp16_tuned_stable-diffusion-v1-4 Downloading (…)tokenizer/vocab.json: 100%|████████████████████████████████████████| 1.06M/1.06M [00:00<00:00, 2.42MB/s] Downloading (…)tokenizer/merges.txt: 100%|██████████████████████████████████████████| 525k/525k [00:00<00:00, 4.40MB/s] Downloading (…)cial_tokens_map.json: 100%|█████████████████████████████████████████████| 472/472 [00:00<00:00, 464kB/s] Downloading (…)okenizer_config.json: 100%|█████████████████████████████████████████████| 806/806 [00:00<00:00, 796kB/s] Downloading (…)_encoder/config.json: 100%|████████████████████████████████████████████████████| 592/592 [00:00<?, ?B/s] Downloading model.safetensors: 100%|████████████████████████████████████████████████| 492M/492M [00:39<00:00, 12.6MB/s] transformers\modeling_utils.py:429: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() with safe_open(checkpoint_file, framework="pt") as f: No vmfb found. Compiling and saving to C:\SHARK Stable Diffusion\SHARK\clip_1_64_512_512_fp16_tuned_stable-diffusion-v1-4_vulkan.vmfb Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args Saved vmfb in C:\SHARK Stable Diffusion\SHARK\clip_1_64_512_512_fp16_tuned_stable-diffusion-v1-4_vulkan.vmfb. WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files. Downloading (…)ain/unet/config.json: 100%|█████████████████████████████████████████████| 743/743 [00:00<00:00, 704kB/s] Downloading (…)ch_model.safetensors: 100%|████████████████████████████████████████| 3.44G/3.44G [04:37<00:00, 12.4MB/s] mat1 and mat2 shapes cannot be multiplied (128x1024 and 768x320) Retrying with a different base model configuration Loading Winograd config file from C:\Users\Acqua.local/shark_tank/configs\unet_winograd_vulkan.json 100%|███████████████████████████████████████████████████████████████████████████████████| 107/107 [00:00<00:00, 618B/s] 100%|█████████████████████████████████████████████████████████████████████████████████| 107/107 [00:00<00:00, 3.00kB/s] Loading lowering config file from C:\Users\Acqua.local/shark_tank/configs\unet_v1_4_fp16_vulkan_rdna2.json 100%|██████████████████████████████████████████████████████████████████████████████| 24.6k/24.6k [00:00<00:00, 125kB/s] 100%|██████████████████████████████████████████████████████████████████████████████| 24.6k/24.6k [00:00<00:00, 440kB/s] Applying tuned configs on unet_1_64_512_512_fp16_tuned_stable-diffusion-v1-4_vulkan No vmfb found. Compiling and saving to C:\SHARK Stable Diffusion\SHARK\unet_1_64_512_512_fp16_tuned_stable-diffusion-v1-4_vulkan.vmfb Using target triple -iree-vulkan-target-triple=rdna2-unknown-windows from command line args Saved vmfb in C:\SHARK Stable Diffusion\SHARK\unet_1_64_512_512_fp16_tuned_stable-diffusion-v1-4_vulkan.vmfb. WARNING: [Loader Message] Code 0 : windows_read_data_files_in_registry: Registry lookup failed to get layer manifest files.