Core dumped with performance counter mode profiling - Githubissues

ROCm / rocprofiler

ROC profiler library. Profiling with perf-counters and derived metrics.

https://rocm.docs.amd.com/projects/rocprofiler/en/latest/

Other

119 stars 45 forks source link

Core dumped with performance counter mode profiling #27

Open syifan opened 4 years ago

syifan commented 4 years ago

I am experiencing a core dumped error when I am running a simple HIP program with the profiler.

I am running the vector add example from the HIP-Examples repo. After making the executable file. I used /opt/rocm/bin/rocprof --stats vectoradd_hip.exe to profile the program. The program cannot properly terminate and I see this error:

/opt/rocm/bin/rocprof: line 275: 17919 Aborted                 (core dumped) "vectoradd_hip.exe"

Here is my machine configuration:

ROCm version: 3.5

"name -a":

Linux lowfreq 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

"lsb_release -a":

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

"rocm-info":

[37mROCk module is loaded[0m
[37mAble to open /dev/kfd read-write[0m
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  Uuid:                    CPU-XX                             
  Marketing Name:          Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3100                               
  BDFID:                   0                                  
  Internal Node ID:        0                                  
  Compute Unit:            20                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    65878732(0x3ed3acc) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    65878732(0x3ed3acc) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
    N/A                      
*******                  
Agent 2                  
*******                  
  Name:                    Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  Uuid:                    CPU-XX                             
  Marketing Name:          Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             CPU                                
  Cache Info:              
    L1:                      32768(0x8000) KB                   
  Chip ID:                 0(0x0)                             
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   3100                               
  BDFID:                   0                                  
  Internal Node ID:        1                                  
  Compute Unit:            20                                 
  SIMDs per CU:            0                                  
  Shader Engines:          0                                  
  Shader Arrs. per Eng.:   0                                  
  WatchPts on Addr. Ranges:1                                  
  Features:                None
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    66033596(0x3ef97bc) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
    Pool 2                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    66033596(0x3ef97bc) KB             
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       TRUE                               
  ISA Info:                
    N/A                      
*******                  
Agent 3                  
*******                  
  Name:                    gfx803                             
  Uuid:                    GPU-XX                             
  Marketing Name:          Fiji [Radeon R9 FURY / NANO Series]
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          4096(0x1000)                       
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    2                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
  Chip ID:                 29440(0x7300)                      
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   1000                               
  BDFID:                   33280                              
  Internal Node ID:        2                                  
  Compute Unit:            64                                 
  SIMDs per CU:            4                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   1                                  
  WatchPts on Addr. Ranges:4                                  
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      FALSE                              
  Wavefront Size:          64(0x40)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        40(0x28)                           
  Max Work-item Per CU:    2560(0xa00)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    4194304(0x400000) KB               
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx803          
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*******                  
Agent 4                  
*******                  
  Name:                    gfx803                             
  Uuid:                    GPU-XX                             
  Marketing Name:          Fiji [Radeon R9 FURY / NANO Series]
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          4096(0x1000)                       
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    3                                  
  Device Type:             GPU                                
  Cache Info:              
    L1:                      16(0x10) KB                        
  Chip ID:                 29440(0x7300)                      
  Cacheline Size:          64(0x40)                           
  Max Clock Freq. (MHz):   1000                               
  BDFID:                   33536                              
  Internal Node ID:        3                                  
  Compute Unit:            64                                 
  SIMDs per CU:            4                                  
  Shader Engines:          4                                  
  Shader Arrs. per Eng.:   1                                  
  WatchPts on Addr. Ranges:4                                  
  Features:                KERNEL_DISPATCH 
  Fast F16 Operation:      FALSE                              
  Wavefront Size:          64(0x40)                           
  Workgroup Max Size:      1024(0x400)                        
  Workgroup Max Size per Dimension:
    x                        1024(0x400)                        
    y                        1024(0x400)                        
    z                        1024(0x400)                        
  Max Waves Per CU:        40(0x28)                           
  Max Work-item Per CU:    2560(0xa00)                        
  Grid Max Size:           4294967295(0xffffffff)             
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)             
    y                        4294967295(0xffffffff)             
    z                        4294967295(0xffffffff)             
  Max fbarriers/Workgrp:   32                                 
  Pool Info:               
    Pool 1                   
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
      Size:                    4194304(0x400000) KB               
      Allocatable:             TRUE                               
      Alloc Granule:           4KB                                
      Alloc Alignment:         4KB                                
      Accessible by all:       FALSE                              
    Pool 2                   
      Segment:                 GROUP                              
      Size:                    64(0x40) KB                        
      Allocatable:             FALSE                              
      Alloc Granule:           0KB                                
      Alloc Alignment:         0KB                                
      Accessible by all:       FALSE                              
  ISA Info:                
    ISA 1                    
      Name:                    amdgcn-amd-amdhsa--gfx803          
      Machine Models:          HSA_MACHINE_MODEL_LARGE            
      Profiles:                HSA_PROFILE_BASE                   
      Default Rounding Mode:   NEAR                               
      Default Rounding Mode:   NEAR                               
      Fast f16:                TRUE                               
      Workgroup Max Size:      1024(0x400)                        
      Workgroup Max Size per Dimension:
        x                        1024(0x400)                        
        y                        1024(0x400)                        
        z                        1024(0x400)                        
      Grid Max Size:           4294967295(0xffffffff)             
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)             
        y                        4294967295(0xffffffff)             
        z                        4294967295(0xffffffff)             
      FBarrier Max Size:       32                                 
*** Done ***

eshcherb commented 4 years ago

Thank you for reporting the issue. I will try to reproduce it.

syifan commented 4 years ago

@eshcherb Any update on this thread? I recently noticed that even simply running rocprof --list-basic can cause the same problem.

eshcherb commented 3 years ago

Abort (core dumped) is exit of profiler on an error. Could you provide complete log with the error message?

syifan commented 3 years ago

Screen Shot 2020-07-23 at 7 03 16 PM

This is the error I get when I simply run --list-basic.

MathiasMagnus commented 3 years ago

I'm seeing the same error when trying to collect perf counters via the CLI tool (rocprof) and also while just trying to list the counters. ROCm 4.0 and the issue is the same. When I'm reading counters from an app that doesn't launch kernels (rocminfo for eg.) it doesn't crash, so at least init and cleanup aren't at fault.

@syifan are you looking for the actual dump file?

eshcherb commented 3 years ago

Hi, it looks like you have corrupted ROCM installation. ROCr runtime cannot find 'libhsa-amd-aqlprofile64.so' library.

Run the profiling with 'LD_DEBUG=libs': $ LD_DEBUG=libs rocprof > log.txt 2>&1

And you can check where it looks 'libhsa-amd-aqlprofile64.so' library. It should be there '/opt/rocm/lib/libhsa-amd-aqlprofile64.so'. If you have it then it might that you didn't update the ROCM properly and you have an old version with incompatible API.

So I would recommend to wipe out your current ROCM completely and install the latest according the instructions from ROCM GitHub page: https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html

eshcherb commented 3 years ago

And please upload the 'log.txt' with the output from 'LD_DEBUG=libs'.

MathiasMagnus commented 3 years ago

@eshcherb It was down to libhsa-amd-aqlprofile64.so missing from the install. Issuing sudo apt install --reinstall hsa-amd-aqlprofile did the trick. Thank you for the pointer.

Even though I work for a company subcontracted by AMD and am fairly well versed in the ways of administering a ROCm install, even I don't know where to report issues regarding the "bigger picture" of installing ROCm libraries. Few things to note:

When looking for the package that provides said missing library, common APT tools can't fetch file information from the repository. If I'm not mistaken, it's Contents.gz missing from the dist, but I may be mistaken here. Something along the lines of this.
```
PS /home/mate> apt search aqlprofile
Sorting... Done
Full Text Search... Done
hsa-amd-aqlprofile/Ubuntu 16.04,now 1.0.0 amd64 [installed]
AQLPROFILE library for AMD HSA runtime API extension support
```

hsa-amd-aqlprofile4.0.0/Ubuntu 16.04 1.0.0 amd64 AQLPROFILE library for AMD HSA runtime API extension support

PS /home/mate> apt-file show hsa-amd-aqlprofile PS /home/mate> apt-file show hsa-amd-aqlprofile4.0.0

Downloading the actual deb file and listing it's contents show that indeed this is the pacakge.

dpkg --contents ./hsa-amd-aqlprofile_1.0.0_amd64.deb drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/ drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/ drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/ drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/ lrwxrwxrwx root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so -> libhsa-amd-aqlprofile64.so.1 lrwxrwxrwx root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so.1 -> libhsa-amd-aqlprofile64.so.1.0.40000 -rw-r--r-- root/root 249304 2020-12-14 12:03 ./opt/rocm-4.0.0/hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so.1.0.40000 drwxr-xr-x root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/lib/ lrwxrwxrwx root/root 0 2020-12-14 12:03 ./opt/rocm-4.0.0/lib/libhsa-amd-aqlprofile64.so -> ../hsa-amd-aqlprofile/lib/libhsa-amd-aqlprofile64.so


- While it is the recommended way of always doing a clean install of ROCm, it's not how users of APT repos have been socialized. Spamming `apt upgrade` and things breaking is generally bad experience. I'm fairly certain nobody went and started deleting libraries inside a ROCm install. The package maintainers should really up their game in terms of test coverage of system upgrades. I'm not sure what's inside the current battery of deployment testing, but changes to the repo contents should likely test upgrade paths from the past 1-2 versions inside docker containers, VMs or anything.

syifan commented 3 years ago

I have been upgrading to ROCm 4.0 and the problem is not there anymore. So I guess @MathiasMagnus is right, it should be an installation issue.