search

ROCm / omnitrace

Omnitrace: Application Profiling, Tracing, and Analysis
https://rocm.docs.amd.com/projects/omnitrace/en/latest/
MIT License
297 stars 27 forks source link

`omnitrace-avail` fails on ROCM 5.3 and RX 6800XT #281

Closed vchuravy closed 1 week ago

vchuravy commented 1 year ago

Using the following docker container:

FROM rocm/rocm-terminal:5.3

RUN sudo apt-get update && \
    sudo apt-get dist-upgrade -y && \
    sudo apt-get install -y wget vim

RUN wget https://github.com/AMDResearch/omnitrace/releases/download/v1.10.0/omnitrace_1.10.0-ubuntu20.04.ROCm50300.PAPI.OMPT.Python3_amd64.deb && \
    sudo dpkg -i omnitrace_1.10.0-ubuntu20.04.ROCm50300.PAPI.OMPT.Python3_amd64.deb && \
    rm *.deb

Running with: docker run -it --device=/dev/kfd --device=/dev/dri --group-add video

*******                  
Agent 2                  
*******                  
  Name:                    gfx1030                            
  Uuid:                    GPU-849e7f8e4045a584               
  Marketing Name:          AMD Radeon RX 6800 XT              
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE   
rocm-user@0010d1870e71:~$ /opt/omnitrace/bin/omnitrace-avail                    
[omnitrace][184][0][fatal] 
[omnitrace][184][0][fatal] ERROR :: rocprofiler_iterate_info(), ImportMetrics(), Bad metric 'L2CacheHit', var 'TCC_HIT[0]' is not found

[omnitrace][184][0] Signal 6 caught : Aborted (Signal sent by tkill() 184 1000)

### ERROR ### [omnitrace][PID=184][TID=0] signal=6 (SIGABRT) abort program (formerly SIGIOT). code: -6
Backtrace:
[PID=184][TID=0][0/9] __restore_rt
[PID=184][TID=0][1/9] gsignal +0xcb
[PID=184][TID=0][2/9] abort +0x12b
[PID=184][TID=0][3/9] _ZN9omnitrace11rocprofiler5fatalERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE.cold +0x5
[PID=184][TID=0][4/9] _ZN9omnitrace11rocprofiler17rocm_check_statusE12hsa_status_tRKSt3setIS1_St4lessIS1_ESaIS1_EE +0xa0
[PID=184][TID=0][5/9] _ZN9omnitrace11rocprofiler12rocm_metricsEv +0x156
[PID=184][TID=0][6/9] main +0x3d47
[PID=184][TID=0][7/9] __libc_start_main +0xf3
[PID=184][TID=0][8/9] _start +0x2e

Backtrace (demangled):
[PID=184][TID=0][0/9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f02b03f1420]
[PID=184][TID=0][1/9] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f02ae30200b]
[PID=184][TID=0][2/9] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f02ae2e1859]
[PID=184][TID=0][3/9] /opt/omnitrace/bin/omnitrace-avail(+0x129aab) [0x55b1fe3d5aab]
[PID=184][TID=0][4/9] /opt/omnitrace/bin/omnitrace-avail(+0x6f99b0) [0x55b1fe9a59b0]
[PID=184][TID=0][5/9] /opt/omnitrace/bin/omnitrace-avail(+0x6fac16) [0x55b1fe9a6c16]
[PID=184][TID=0][6/9] /opt/omnitrace/bin/omnitrace-avail(+0x4b87b7) [0x55b1fe7647b7]
[PID=184][TID=0][7/9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f02ae2e3083]
[PID=184][TID=0][8/9] /opt/omnitrace/bin/omnitrace-avail(+0x4f3b8e) [0x55b1fe79fb8e]

/proc/184/maps:
    55b1fe2ac000-55b1fe3cb000 r--p 00000000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b1fe3cb000-55b20030f000 r-xp 0011f000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b20030f000-55b20086a000 r--p 02063000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b20086b000-55b2008b2000 r--p 025be000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b2008b2000-55b20092b000 rw-p 02605000 00:18 40421                      /opt/omnitrace/bin/omnitrace-avail
    55b20092b000-55b200f5d000 rw-p 00000000 00:00 0 
    55b201875000-55b20250a000 rw-p 00000000 00:00 0                          [heap]
    7f0197a00000-7f0198241000 r--p 00000000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f0198241000-7f019d738000 r-xp 00841000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019d738000-7f019f908000 r--p 05d38000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019f908000-7f019f909000 ---p 07f08000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019f909000-7f019fe4c000 r--p 07f08000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019fe4c000-7f019fe9c000 rw-p 0844b000 00:18 4529                       /opt/rocm-5.3.0/lib/libamd_comgr.so.2.4.50300
    7f019fe9c000-7f019fefc000 rw-p 00000000 00:00 0 
    7f01a0000000-7f01a0021000 rw-p 00000000 00:00 0 
    7f01a0021000-7f01a4000000 ---p 00000000 00:00 0 
    7f01a5800000-7f02a5800000 ---p 00000000 00:00 0 
    7f02a6cae000-7f02a8000000 rw-p 00000000 00:00 0 
    7f02a8000000-7f02a8021000 rw-p 00000000 00:00 0 
    7f02a8021000-7f02ac000000 ---p 00000000 00:00 0 
    7f02ac59c000-7f02acb76000 rw-p 00000000 00:00 0 
    7f02acb76000-7f02acc07000 rw-p 00000000 00:00 0 
    7f02acc67000-7f02acca7000 rw-p 00000000 00:00 0 
    7f02acca7000-7f02accb1000 r--p 00000000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accb1000-7f02acce4000 r-xp 0000a000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02acce4000-7f02accef000 r--p 0003d000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accef000-7f02accf0000 ---p 00048000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accf0000-7f02accf3000 r--p 00048000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02accf3000-7f02acd00000 rw-p 0004b000 00:18 4542                       /opt/rocm-5.3.0/lib/libhsa-amd-aqlprofile64.so.1.0.50300
    7f02acd00000-7f02ace01000 rw-p 00000000 00:00 0 
    7f02ace09000-7f02ace3a000 rw-p 00000000 00:00 0 
    7f02ace6b000-7f02acf00000 rw-p 00000000 00:00 0 
    7f02acf00000-7f02ad001000 rw-p 00000000 00:00 0 
    7f02ad015000-7f02ad05b000 rw-p 00000000 00:00 0 
    7f02ad05b000-7f02ad069000 r--p 00000000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad069000-7f02ad078000 r-xp 0000e000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad078000-7f02ad086000 r--p 0001d000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad086000-7f02ad08a000 r--p 0002a000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad08a000-7f02ad08b000 rw-p 0002e000 00:18 1307                       /usr/lib/x86_64-linux-gnu/libtinfo.so.6.2
    7f02ad08b000-7f02ad08c000 ---p 00000000 00:00 0 
    7f02ad08c000-7f02ad88c000 rwxp 00000000 00:00 0 
    7f02ad88c000-7f02ad88d000 ---p 00000000 00:00 0 
    7f02ad88d000-7f02ae08d000 rwxp 00000000 00:00 0 
    7f02ae08d000-7f02ae0dd000 rw-p 00000000 00:00 0 
    7f02ae0dd000-7f02ae173000 r--p 00000000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae173000-7f02ae264000 r-xp 00096000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae264000-7f02ae2ad000 r--p 00187000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2ad000-7f02ae2ae000 ---p 001d0000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2ae000-7f02ae2b9000 r--p 001d0000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2b9000-7f02ae2bc000 rw-p 001db000 00:18 1297                       /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
    7f02ae2bc000-7f02ae2bf000 rw-p 00000000 00:00 0 
    7f02ae2bf000-7f02ae2e1000 r--p 00000000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae2e1000-7f02ae459000 r-xp 00022000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae459000-7f02ae4a7000 r--p 0019a000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae4a7000-7f02ae4ab000 r--p 001e7000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae4ab000-7f02ae4ad000 rw-p 001eb000 00:18 1187                       /usr/lib/x86_64-linux-gnu/libc-2.31.so
    7f02ae4ad000-7f02ae4b1000 rw-p 00000000 00:00 0 
    7f02ae4b1000-7f02ae4be000 r--p 00000000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae4be000-7f02ae565000 r-xp 0000d000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae565000-7f02ae5fe000 r--p 000b4000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae5fe000-7f02ae5ff000 r--p 0014c000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae5ff000-7f02ae600000 rw-p 0014d000 00:18 1229                       /usr/lib/x86_64-linux-gnu/libm-2.31.so
    7f02ae600000-7f02ae61e000 r--p 00000000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02ae61e000-7f02ae99e000 r-xp 0001e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02ae99e000-7f02aea8e000 r--p 0039e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02aea8e000-7f02aea8f000 ---p 0048e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02aea8f000-7f02aea95000 r--p 0048e000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02aea95000-7f02affc9000 rw-p 00494000 00:18 4532                       /opt/rocm-5.3.0/lib/libamdhip64.so.5.3.50300
    7f02affc9000-7f02affda000 rw-p 00000000 00:00 0 
    7f02affe6000-7f02affe7000 rw-p 00000000 00:00 0 
    7f02affe8000-7f02afff0000 rw-s 107ae6000 00:b8 15                        /dev/dri/renderD128
    7f02afff2000-7f02afff4000 rw-p 00000000 00:00 0 
    7f02afff5000-7f02afff6000 rw-p 00000000 00:00 0 
    7f02afff7000-7f02b0000000 rw-p 00000000 00:00 0 
    7f02b0000000-7f02b00ef000 r--p 00000000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b00ef000-7f02b0100000 r-xp 000ef000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b0100000-7f02b0193000 r--p 00100000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b0193000-7f02b0194000 ---p 00193000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b0194000-7f02b023c000 r--p 00193000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b023c000-7f02b029d000 rw-p 0023b000 00:18 40673                      /opt/omnitrace/lib/omnitrace/libpfm.so.4.11.1
    7f02b029d000-7f02b029f000 rw-p 00000000 00:00 0 
    7f02b029f000-7f02b02a4000 rw-p 00000000 00:00 0 
    7f02b02a4000-7f02b02a7000 r--p 00000000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02a7000-7f02b02ac000 r-xp 00003000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02ac000-7f02b02ae000 r--p 00008000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02ae000-7f02b02af000 ---p 0000a000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02af000-7f02b02b0000 r--p 0000a000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02b0000-7f02b02b1000 rw-p 0000b000 00:18 19012                      /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
    7f02b02b1000-7f02b02b4000 r--p 00000000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02b4000-7f02b02bb000 r-xp 00003000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02bb000-7f02b02bd000 r--p 0000a000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02bd000-7f02b02be000 r--p 0000b000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02be000-7f02b02bf000 rw-p 0000c000 00:18 3907                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1.0.0
    7f02b02bf000-7f02b02c4000 r--p 00000000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02c4000-7f02b02d1000 r-xp 00005000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d1000-7f02b02d5000 r--p 00012000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d5000-7f02b02d6000 r--p 00015000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d6000-7f02b02d7000 rw-p 00016000 00:18 3905                       /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2.4.0
    7f02b02d7000-7f02b02da000 r--p 00000000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02da000-7f02b02ec000 r-xp 00003000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02ec000-7f02b02f1000 r--p 00015000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02f1000-7f02b02f2000 r--p 00019000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02f2000-7f02b02f3000 rw-p 0001a000 00:18 18872                      /usr/lib/x86_64-linux-gnu/libelf-0.176.so
    7f02b02f3000-7f02b02f5000 rw-p 00000000 00:00 0 
    7f02b02f5000-7f02b02f8000 r--p 00000000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b02f8000-7f02b030a000 r-xp 00003000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b030a000-7f02b030e000 r--p 00015000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b030e000-7f02b030f000 r--p 00018000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b030f000-7f02b0310000 rw-p 00019000 00:18 1212                       /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
    7f02b0310000-7f02b032d000 r--p 00000000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b032d000-7f02b038d000 r-xp 0001d000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b038d000-7f02b03a2000 r--p 0007d000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b03a2000-7f02b03a4000 r--p 00091000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b03a4000-7f02b03a5000 rw-p 00093000 00:18 4560                       /opt/rocm-5.3.0/lib/librocm_smi64.so.5.0.50300
    7f02b03a5000-7f02b03a7000 r--p 00000000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03a7000-7f02b03b2000 r-xp 00002000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b2000-7f02b03b5000 r--p 0000d000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b5000-7f02b03b6000 r--p 0000f000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b6000-7f02b03b7000 rw-p 00010000 00:18 40701                      /opt/omnitrace/lib/omnitrace/libunwind.so.99.0.0
    7f02b03b7000-7f02b03c1000 rw-p 00000000 00:00 0 
    7f02b03c1000-7f02b03c3000 r--p 00000000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03c3000-7f02b03d4000 r-xp 00002000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03d4000-7f02b03da000 r--p 00013000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03da000-7f02b03db000 ---p 00019000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03db000-7f02b03dc000 r--p 00019000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03dc000-7f02b03dd000 rw-p 0001a000 00:18 33853                      /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    7f02b03dd000-7f02b03e3000 r--p 00000000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03e3000-7f02b03f4000 r-xp 00006000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03f4000-7f02b03fa000 r--p 00017000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03fa000-7f02b03fb000 r--p 0001c000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03fb000-7f02b03fc000 rw-p 0001d000 00:18 1281                       /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    7f02b03fc000-7f02b0400000 rw-p 00000000 00:00 0 
    7f02b0400000-7f02b0418000 r--p 00000000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0418000-7f02b04ef000 r-xp 00018000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b04ef000-7f02b0558000 r--p 000ef000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0558000-7f02b0559000 ---p 00158000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0559000-7f02b0561000 r--p 00158000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b0561000-7f02b069c000 rw-p 00160000 00:18 4545                       /opt/rocm-5.3.0/lib/libhsa-runtime64.so.1.7.50300
    7f02b069c000-7f02b06a1000 rw-p 00000000 00:00 0 
    7f02b06a1000-7f02b06a2000 r--p 00000000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06a2000-7f02b06a8000 r-xp 00001000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06a8000-7f02b06ab000 r--p 00007000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06ab000-7f02b06ac000 r--p 00009000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06ac000-7f02b06ad000 rw-p 0000a000 00:18 40657                      /opt/omnitrace/lib/omnitrace/libgotcha.so.2.0.2
    7f02b06ad000-7f02b06af000 rw-p 00000000 00:00 0 
    7f02b06af000-7f02b06b1000 r--p 00000000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b1000-7f02b06b5000 r-xp 00002000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b5000-7f02b06b7000 r--p 00006000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b7000-7f02b06b8000 r--p 00007000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b8000-7f02b06b9000 rw-p 00008000 00:18 1285                       /usr/lib/x86_64-linux-gnu/librt-2.31.so
    7f02b06b9000-7f02b06ba000 r--p 00000000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06ba000-7f02b06bc000 r-xp 00001000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06bc000-7f02b06bd000 r--p 00003000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06bd000-7f02b06be000 r--p 00003000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06be000-7f02b06bf000 rw-p 00004000 00:18 1198                       /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    7f02b06bf000-7f02b06c6000 r--p 00000000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b06c6000-7f02b0700000 r-xp 00007000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b0700000-7f02b070d000 r--p 00041000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b070d000-7f02b070e000 ---p 0004e000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b070e000-7f02b070f000 r--p 0004e000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b070f000-7f02b0710000 rw-p 0004f000 00:18 4563                       /opt/rocm-5.3.0/lib/librocprofiler64.so.1.0.50300
    7f02b0710000-7f02b0715000 r--p 00000000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0715000-7f02b0742000 r-xp 00005000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0742000-7f02b0750000 r--p 00032000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0750000-7f02b0751000 r--p 0003f000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0751000-7f02b0752000 rw-p 00040000 00:18 4566                       /opt/rocm-5.3.0/lib/libroctracer64.so.4.1.0
    7f02b0752000-7f02b0756000 rw-p 00000000 00:00 0 
    7f02b0756000-7f02b0757000 rw-s 00000000 00:bc 4                          /dev/shm/rocm_smi_card0
    7f02b0757000-7f02b0758000 rw-p 00000000 00:00 0 
    7f02b0758000-7f02b0759000 rw-p 00000000 00:00 0 
    7f02b0759000-7f02b075a000 rw-s 3e64800000000000 00:b8 11                 /dev/kfd
    7f02b075a000-7f02b075b000 rw-p 00000000 00:00 0 
    7f02b075b000-7f02b075c000 rw-s 00000000 00:bc 3                          /dev/shm/hsakmt_shared_mem
    7f02b075c000-7f02b075d000 r--p 00000000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b075d000-7f02b0780000 r-xp 00001000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b0780000-7f02b0788000 r--p 00024000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b0788000-7f02b0789000 rw-s 00000000 00:bc 2                          /dev/shm/sem.hsakmt_semaphore
    7f02b0789000-7f02b078a000 r--p 0002c000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b078a000-7f02b078b000 rw-p 0002d000 00:18 1165                       /usr/lib/x86_64-linux-gnu/ld-2.31.so
    7f02b078b000-7f02b078c000 rw-p 00000000 00:00 0 
    7ffe321ac000-7ffe321ee000 rwxp 00000000 00:00 0                          [stack]
    7ffe321ee000-7ffe321f0000 rw-p 00000000 00:00 0 
    7ffe321f3000-7ffe321f7000 r--p 00000000 00:00 0                          [vvar]
    7ffe321f7000-7ffe321f9000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

Backtrace (demangled):
[PID=184][TID=0][0/9] __restore_rt
[PID=184][TID=0][1/9] gsignal +0xcb
[PID=184][TID=0][2/9] abort +0x12b
[PID=184][TID=0][3/9] omnitrace::rocprofiler::fatal(std::string const&) [clone .cold] +0x5
[PID=184][TID=0][4/9] omnitrace::rocprofiler::rocm_check_status(hsa_status_t, std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>> const&) +0xa0
[PID=184][TID=0][5/9] omnitrace::rocprofiler::rocm_metrics() +0x156
[PID=184][TID=0][6/9] main +0x3d47
[PID=184][TID=0][7/9] __libc_start_main +0xf3
[PID=184][TID=0][8/9] _start +0x2e

Backtrace (lineinfo):
[PID=184][TID=0][0/7]
    [/lib/x86_64-linux-gnu/libpthread.so.0:?] __restore_rt
[PID=184][TID=0][1/7]
    [/usr/lib/x86_64-linux-gnu/libc-2.31.so:?] abort
[PID=184][TID=0][2/7]
    [/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:75] omnitrace::rocprofiler::fatal(std::string const&)
[PID=184][TID=0][3/7]
    [/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:100] omnitrace::rocprofiler::rocm_check_status(hsa_status_t, std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>> const&)
[PID=184][TID=0][4/7]
    [/home/omnitrace/source/lib/omnitrace/library/rocprofiler.cpp:370] omnitrace::rocprofiler::rocm_metrics()
    [/usr/include/c++/9/bits/stl_set.h:281] std::set<hsa_status_t, std::less<hsa_status_t>, std::allocator<hsa_status_t>>::~set()
    [/usr/include/c++/9/bits/stl_tree.h:1000] std::_Rb_tree<hsa_status_t, hsa_status_t, std::_Identity<hsa_status_t>, std::less<hsa_status_t>, std::allocator<hsa_status_t>>::~_Rb_tree()
[PID=184][TID=0][5/7]
    [/home/omnitrace/source/bin/omnitrace-avail/avail.cpp:513] main
    [/usr/include/c++/9/bits/stl_vector.h:916] std::vector<tim::hardware_counters::info, std::allocator<tim::hardware_counters::info>>::size() const
[PID=184][TID=0][6/7]
    [/usr/lib/x86_64-linux-gnu/libc-2.31.so:?] __libc_start_main

[omnitrace][184] Finalizing afer signal 6 ::  Signal:    SIGABRT (signal number:   6)          abort program (formerly SIGIOT)

[omnitrace][184] Killing process 184 with signal 6...
vchuravy commented 1 year ago

This works when switching to ROCM 5.4

FROM rocm/rocm-terminal:5.4

RUN sudo apt-get update && \
    sudo apt-get dist-upgrade -y && \
    sudo apt-get install -y wget vim

RUN wget https://github.com/AMDResearch/omnitrace/releases/download/v1.10.0/omnitrace_1.10.0-ubuntu20.04.ROCm50400.PAPI.OMPT.Python3_amd64.deb && \
    sudo dpkg -i omnitrace_1.10.0-ubuntu20.04.ROCm50400.PAPI.OMPT.Python3_amd64.deb && \
    rm *.deb
jrmadsen commented 1 year ago

FYI, this is probably a rocprofiler bug, not an omnitrace bug. There have been a lot of issues with its stability between minor releases recently.