search

l1na-forever / stable-diffusion-rocm-docker

Stable Diffusion Docker image preconfigured for usage with AMD Radeon cards
120 stars 20 forks source link

hipErrorNoBinaryForGpu: Unable to find code object for all current devices! #1

Closed ghost closed 1 year ago

ghost commented 1 year ago

I followed the instructions and tried running this, but there seems to be an issue:

docker start -ai stable-diffusion    
    active environment : ldm  
   active env location : /opt/conda/envs/ldm  
Installing PyTorch ROCM5.1.1 version..  
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/rocm5.1.1  
Requirement already satisfied: torch in /opt/conda/envs/ldm/lib/python3.8/site-packages (1.12.1+rocm5.1.1)  
Requirement already satisfied: torchvision in /opt/conda/envs/ldm/lib/python3.8/site-packages (0.13.1+rocm5.1.1)  
Requirement already satisfied: typing-extensions in /opt/conda/envs/ldm/lib/python3.8/site-packages (from torch) (4.3.0)  
Requirement already satisfied: requests in /opt/conda/envs/ldm/lib/python3.8/site-packages (from torchvision) (2.25.1)  
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/ldm/lib/python3.8/site-packages (from torchvision) (9.2.0)  
Requirement already satisfied: numpy in /opt/conda/envs/ldm/lib/python3.8/site-packages (from torchvision) (1.22.3)  
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/ldm/lib/python3.8/site-packages (from requests->torchvision) (2.10)  
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/envs/ldm/lib/python3.8/site-packages (from requests->torchvision) (4.0.0)  
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/envs/ldm/lib/python3.8/site-packages (from requests->torchvision) (1.26.11)  
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/ldm/lib/python3.8/site-packages (from requests->torchvision) (2022.9.24)  
Done installing PyTorch ROCM  
Validating model files...  
checking model.ckpt...  
model.ckpt is valid!  

checking GFPGANv1.3.pth...  
GFPGANv1.3.pth is valid!  

checking RealESRGAN_x4plus.pth...  
RealESRGAN_x4plus.pth is valid!  

checking RealESRGAN_x4plus_anime_6B.pth...  
RealESRGAN_x4plus_anime_6B.pth is valid!  

checking project.yaml...  
project.yaml is valid!  

checking model.ckpt...  
model.ckpt is valid!  

entrypoint.sh: Run python scripts/webui.py ...  
"hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"  
/sd/entrypoint.sh: line 133:    46 Aborted                 (core dumped) $launch_command  
entrypoint.sh: Process is ending. Relaunching in 0.5s...  
entrypoint.sh: Run python scripts/webui.py ...  
Relaunch count: 1  
"hipErrorNoBinaryForGpu: Unable to find code object for all current devices!"  
/sd/entrypoint.sh: line 133:    51 Aborted                 (core dumped) $launch_command  
entrypoint.sh: Process is ending. Relaunching in 0.5s...  
entrypoint.sh: Run python scripts/webui.py ...  
Relaunch count: 2  
^C
Traceback (most recent call last):  
 File "scripts/webui.py", line 6, in <module>  
   from frontend.frontend import draw_gradio_ui  
 File "/sd/frontend/frontend.py", line 6, in <module>  
   import torch  
 File "/opt/conda/envs/ldm/lib/python3.8/site-packages/torch/__init__.py", line 811, in <module>  
   from .functional import *  # noqa: F403  
 File "/opt/conda/envs/ldm/lib/python3.8/site-packages/torch/functional.py", line 7, in <module>  
   import torch.nn.functional as F  
 File "/opt/conda/envs/ldm/lib/python3.8/site-packages/torch/nn/__init__.py", line 1, in <module>  
   from .modules import *  # noqa: F403  
 File "/opt/conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/__init__.py", line 2, in <module>  
   from .linear import Identity, Linear, Bilinear, LazyLinear  
 File "/opt/conda/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 7, in <module>  
   from .. import functional as F  
 File "/opt/conda/envs/ldm/lib/python3.8/site-packages/torch/nn/functional.py", line 18, in <module>  
   from .._jit_internal import boolean_dispatch, _overload, BroadcastingList1, BroadcastingList2, BroadcastingList3  
 File "<frozen importlib._bootstrap>", line 991, in _find_and_load  
 File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked  
 File "<frozen importlib._bootstrap>", line 671, in _load_unlocked  
 File "<frozen importlib._bootstrap_external>", line 779, in exec_module  
 File "<frozen importlib._bootstrap_external>", line 911, in get_code  
 File "<frozen importlib._bootstrap_external>", line 580, in _compile_bytecode  
KeyboardInterrupt

relevant pc specs:

OS: Artix Linux x86_64 
Kernel: 5.19.13-zen1-1-zen 
CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz 
GPU: AMD ATI Radeon RX 6600/6600 XT/6600M 
Memory: 3983MiB / 32004MiB

rocm seems to be working 

rocminfo

ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 7 5800X 8-Core Processor 
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 7 5800X 8-Core Processor 
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3800                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            16                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: FINE GRAINED        
      Size:                    32772280(0x1f410b8) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    32772280(0x1f410b8) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 3                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    32772280(0x1f410b8) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
*******                  
Agent 2                  
*******                  
  Name:                    gfx1032                            
  Uuid:                    GPU-XX                             
  Marketing Name:          AMD Radeon RX 6600 XT              
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
    L2:                      2048(0x800) KB                     
    L3:                      32768(0x8000) KB                   
  Chip ID:                 29695(0x73ff)                      
  ASIC Revision:           0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   2900                               
  BDFID:                   3328                               
  Internal Node ID:        1                                  
  Compute Unit:            32                                 
  SIMDs per CU:            2                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   2                                  
  WatchPts on Addr. Ranges:4                                  
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      TRUE                               
  Wavefront Size:          32(0x20)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        32(0x20)                           
  Max Work-item Per CU:    1024(0x400)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    8372224(0x7fc000) KB               
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx1032         
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***

Did I miss some configuration? If there is anything I can try out or any details to report back, please let me know.

Thank you very much for your work!

l1na-forever commented 1 year ago

Hi there! Thanks for all the detail in the report :) I'm far from an expert on the ROCM stuff, but it looks like the HIP libraries in the docker image don't think they have the right kernel for your card's arch (gfx1032). From what I can tell, there's not necessarily a large/any? difference between kernels for gfx103x series, since our cards are all at the same compute feature level (GFX 10.3). I'm wondering if all it would take is the HSA_OVERRIDE_GFX_VERSION=10.3.0 flag set within your docker container.

Could you try adding -e HSA_OVERRIDE_GFX_VERSION=10.3.0 to your docker run invocation?

ghost commented 1 year ago

Awesome that did the trick!

Thank you very much for your help :)