Open geohot opened 1 year ago
massivedynamics1
commented
1 year ago Thanks for connecting @realGeorgeHotz. Appreciate the work you and tiny corp are doing. We are committed to working with the community and improving our support. More to come on ROCm on @radeon soon. Lots of work ahead but excited about what we can do together.
— Lisa Su (@LisaSu) June 16, 2023
CorvetteCole
commented
10 months ago I can reproduce this on my 7900 XTX as well
LefterisJP
commented
10 months ago I am also hitting this
aenertia
commented
9 months ago I am seeing similar messages from kernel on boot with phoenix apu based system and without the ROCM Stack
1637.604718] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14 [ 1637.605055] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait [ 1637.607290] [drm] PCIE GART of 512M enabled (table at 0x00000080FFD00000). [ 1637.607422] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming... [ 1637.610838] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully! [ 1637.803597] [drm] VCN decode and encode initialized successfully(under DPG Mode). [ 1637.803867] amdgpu 0000:c1:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully. [ 1637.804227] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0 [ 1637.804231] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0 [ 1637.804234] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0 [ 1637.804235] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0 [ 1637.804237] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0 [ 1637.804239] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0 [ 1637.804241] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0 [ 1637.804243] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0 [ 1637.804245] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0 [ 1637.804246] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0 [ 1637.804248] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8 [ 1637.804250] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8 [ 1637.804252] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0 [ 1637.809886] [drm] ring gfx_32803.1.1 was added [ 1637.810426] [drm] ring compute_32803.2.2 was added [ 1637.810934] [drm] ring sdma_32803.3.3 was added [ 1637.810962] [drm] ring gfx_32803.1.1 ib test pass [ 1637.810991] [drm] ring compute_32803.2.2 ib test pass [ 1637.811092] [drm] ring sdma_32803.3.3 ib test pass [ 1637.820482] PM: resume devices took 0.362 seconds [ 1637.820898] OOM killer enabled. [ 1637.820902] Restarting tasks ... done. [ 1637.822675] random: crng reseeded on system resumption [ 1637.829255] PM: suspend exit [ 1640.456439] wlp1s0: authenticate with 24:4b:fe:62:67:04 (local address=14:ac:60:46:7b:f9) [ 1640.479251] wlp1s0: send auth to 24:4b:fe:62:67:04 (try 1/3) [ 1640.499491] wlp1s0: authenticate with 24:4b:fe:62:67:04 (local address=14:ac:60:46:7b:f9) [ 1640.508904] wlp1s0: send auth to 24:4b:fe:62:67:04 (try 1/3) [ 1640.512391] wlp1s0: authenticated [ 1640.515427] wlp1s0: associate with 24:4b:fe:62:67:04 (try 1/3) [ 1640.526627] wlp1s0: RX AssocResp from 24:4b:fe:62:67:04 (capab=0x1511 status=0 aid=1) [ 1640.558571] wlp1s0: associated [ 1640.613407] wlp1s0: Limiting TX power to 27 (30 - 3) dBm as advertised by 24:4b:fe:62:67:04 [ 1642.696635] ucsi_acpi USBC000:00: ucsi_handle_connector_change: ACK failed (-110) [ 1648.577999] ucsi_acpi USBC000:00: ucsi_handle_connector_change: ACK failed (-110) [ 2325.112034] i2c_designware AMDI0010:00: i2c_dw_handle_tx_abort: lost arbitration [ 2531.128560] wlp1s0: deauthenticating from 24:4b:fe:62:67:04 by local choice (Reason: 3=DEAUTH_LEAVING) [ 2531.678155] PM: suspend entry (s2idle) [ 2531.701397] Filesystems sync: 0.023 seconds [ 2531.710174] Freezing user space processes [ 2531.712040] Freezing user space processes completed (elapsed 0.001 seconds) [ 2531.712044] OOM killer disabled. [ 2531.712045] Freezing remaining freezable tasks [ 2531.713066] Freezing remaining freezable tasks completed (elapsed 0.001 seconds) [ 2531.713070] printk: Suspending console(s) (use no_console_suspend to debug) [ 2532.645147] PM: suspend devices took 0.932 seconds [ 2532.647493] ACPI: EC: interrupt blocked [ 9429.864695] ACPI: EC: interrupt unblocked [ 9430.036343] nvme nvme0: 16/0/0 default/read/poll queues [ 9430.088770] atkbd serio0: Unknown key pressed (translated set 2, code 0x6b on isa0060/serio0). [ 9430.088779] atkbd serio0: Use 'setkeycodes 6b <keycode>' to make it known. [ 9430.089328] atkbd serio0: Unknown key released (translated set 2, code 0x6b on isa0060/serio0). [ 9430.089336] atkbd serio0: Use 'setkeycodes 6b <keycode>' to make it known. [ 9430.175277] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14 [ 9430.175614] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait [ 9430.177835] [drm] PCIE GART of 512M enabled (table at 0x00000080FFD00000). [ 9430.177942] amdgpu 0000:c1:00.0: amdgpu: SMU is resuming... [ 9430.181299] amdgpu 0000:c1:00.0: amdgpu: SMU is resumed successfully! [ 9430.286638] [drm] VCN decode and encode initialized successfully(under DPG Mode). [ 9430.286884] amdgpu 0000:c1:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully. [ 9430.287225] amdgpu 0000:c1:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0 [ 9430.287229] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0 [ 9430.287231] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0 [ 9430.287233] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0 [ 9430.287234] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0 [ 9430.287236] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0 [ 9430.287237] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0 [ 9430.287239] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0 [ 9430.287241] amdgpu 0000:c1:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0 [ 9430.287242] amdgpu 0000:c1:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0 [ 9430.287244] amdgpu 0000:c1:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 8 [ 9430.287246] amdgpu 0000:c1:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 8 [ 9430.287247] amdgpu 0000:c1:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0 [ 9430.292387] [drm] ring gfx_32803.1.1 was added [ 9430.292929] [drm] ring compute_32803.2.2 was added [ 9430.293440] [drm] ring sdma_32803.3.3 was added [ 9430.293469] [drm] ring gfx_32803.1.1 ib test pass [ 9430.293499] [drm] ring compute_32803.2.2 ib test pass [ 9430.293598] [drm] ring sdma_32803.3.3 ib test pass
💻 Framework Laptop 13 (AMD Ryzen 7040Series) (Laptop) running BIOS 3.3 (03.03) released 10/17/2023 and EC unknown
🐧 Fedora Linux 40 (KDE Plasma Prerelease)
🐧 Kernel 6.7.0fw13ec-rc1+
✅ AMD Ryzen 7 7840U w/ Radeon 780M Graphics (family 19 model 74)
✅ HSMP driver amd_hsmp not detected (blocked: False)
✅ PMC driver amd_pmc loaded (Program 0 Firmware 76.70.0)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 24.0.0-devel
excieve
commented
9 months ago Similar issue with 7900 XT, ROCm 5.7, kernel 6.5.11 (Fedora 39)
Although GPU recovery doesn't fully help it seems. While there are no MES errors and GPU stats (usage, power draw) seem to be back to normal, there's a lot of the following in the kernel logs:
[drm] Skip scheduling IBs!
Which keeps happening until a reboot.
kean0048
commented
9 months ago I have got the same msg during suspend the device based on RHEL9.3 which kernel is 'Linux localhost.localdomain 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux'
hardware info: 1) VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev d3) (prog-if 00 [VGA controller]) 2) 3D controller [0302]: NVIDIA Corporation AD107GLM [RTX 2000 Ada Generation Laptop GPU] [10de:28b8] (rev a1)
[ 232.371283] OOM killer disabled. [ 232.371284] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done. [ 232.372385] printk: Suspending console(s) (use no_console_suspend to debug) [ 232.846688] ACPI: EC: interrupt blocked [ 233.975847] ACPI: EC: interrupt unblocked [ 234.994441] nvme nvme0: 16/0/0 default/read/poll queues [ 234.995443] nvme nvme1: 16/0/0 default/read/poll queues [ 235.106268] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] ERROR MES failed to response msg=14 [ 235.106526] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] ERROR failed to reg_write_reg_wait [ 235.108935] [drm] PCIE GART of 512M enabled (table at 0x000000803FD00000). [ 235.109033] amdgpu 0000:c6:00.0: amdgpu: SMU is resuming... [ 235.112006] amdgpu 0000:c6:00.0: amdgpu: SMU is resumed successfully! [ 235.114265] [drm] Watermarks table not configured properly by SMU [ 235.167424] [drm] VCN decode and encode initialized successfully(under DPG Mode). [ 235.167510] amdgpu 0000:c6:00.0: [drm:jpeg_v4_0_hw_init [amdgpu]] JPEG decode initialized successfully. [ 235.167631] amdgpu 0000:c6:00.0: amdgpu: ring gfx_0.0.0 uses VM inv eng 0 on hub 0 [ 235.167632] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.0.0 uses VM inv eng 1 on hub 0 [ 235.167632] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.1.0 uses VM inv eng 4 on hub 0 [ 235.167633] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.2.0 uses VM inv eng 6 on hub 0 [ 235.167634] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.3.0 uses VM inv eng 7 on hub 0 [ 235.167634] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.0.1 uses VM inv eng 8 on hub 0 [ 235.167634] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.1.1 uses VM inv eng 9 on hub 0 [ 235.167635] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.2.1 uses VM inv eng 10 on hub 0 [ 235.167635] amdgpu 0000:c6:00.0: amdgpu: ring comp_1.3.1 uses VM inv eng 11 on hub 0 [ 235.167636] amdgpu 0000:c6:00.0: amdgpu: ring sdma0 uses VM inv eng 12 on hub 0 [ 235.167636] amdgpu 0000:c6:00.0: amdgpu: ring vcn_unified_0 uses VM inv eng 0 on hub 1 [ 235.167637] amdgpu 0000:c6:00.0: amdgpu: ring jpeg_dec uses VM inv eng 1 on hub 1 [ 235.167637] amdgpu 0000:c6:00.0: amdgpu: ring mes_kiq_3.1.0 uses VM inv eng 13 on hub 0 [ 235.173037] [drm] ring gfx_32770.1.1 was added [ 235.173947] [drm] ring compute_32770.2.2 was added [ 235.174822] [drm] ring sdma_32770.3.3 was added [ 235.174828] [drm] ring gfx_32770.1.1 test pass [ 235.174845] [drm] ring gfx_32770.1.1 ib test pass [ 235.174850] [drm] ring compute_32770.2.2 test pass [ 235.174871] [drm] ring compute_32770.2.2 ib test pass [ 235.175007] [drm] ring sdma_32770.3.3 test pass [ 235.175039] [drm] ring sdma_32770.3.3 ib test pass [ 235.312399] OOM killer enabled.
and suspend failed.
terryrankine
commented
9 months ago Dec 05 21:26:14 theblob kernel: amdgpu: failed to remove hardware queue from MES, doorbell=0x1002
Dec 05 21:26:14 theblob kernel: amdgpu: MES might be in unrecoverable state, issue a GPU reset
Dec 05 21:26:14 theblob kernel: amdgpu: Failed to evict queue 1
Dec 05 21:26:14 theblob kernel: amdgpu: Failed to evict process queues
Dec 05 21:26:14 theblob kernel: amdgpu: Failed to evict queues of pasid 0x8001
Dec 05 21:26:14 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: GPU reset begin!
Dec 05 21:26:14 theblob google-chrome.desktop[11352]: [21438:21449:1205/212614.765940:ERROR:vulkan_swap_chain.cc(443)] vkAcquireNextImageKHR() hangs.
Dec 05 21:26:14 theblob google-chrome.desktop[11352]: [21438:21438:1205/212614.766188:ERROR:gpu_service_impl.cc(1105)] Exiting GPU process because some drivers can't recover from errors. GPU process will restart shortly.
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: IP block:gfx_v11_0 is hung!
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa00200 flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa002b4 flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa002f4 flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa00208 flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa00200 flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa0025c flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa0029c flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa002dc flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa00208 flags=0x0020]
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x001e address=0xffc5fa00200 flags=0x0020]
Dec 05 21:26:15 theblob kernel: [drm] kiq ring mec 3 pipe 1 q 0
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: recover vram bo from shadow start
Dec 05 21:26:15 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: recover vram bo from shadow done
Dec 05 21:26:15 theblob google-chrome.desktop[11352]: [11346:11346:1205/212615.493041:ERROR:gpu_process_host.cc(990)] GPU process exited unexpectedly: exit_code=8704
Dec 05 21:26:15 theblob kernel: [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
Dec 05 21:26:15 theblob kernel: [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_waitsnip
Dec 05 21:26:17 theblob kernel: [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
Dec 05 21:26:17 theblob kernel: [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=2
Dec 05 21:26:17 theblob kernel: [drm:amdgpu_mes_add_hw_queue [amdgpu]] *ERROR* failed to add hardware queue to MES, doorbell=0x1a00
Dec 05 21:26:17 theblob kernel: [drm:amdgpu_mes_self_test [amdgpu]] *ERROR* failed to add ring
Dec 05 21:26:17 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: GPU reset(1) succeeded!
Dec 05 21:26:31 theblob kernel: [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
Dec 05 21:26:31 theblob kernel: amdgpu: failed to remove hardware queue from MES, doorbell=0x1000
Dec 05 21:26:31 theblob kernel: amdgpu: MES might be in unrecoverable state, issue a GPU reset
Dec 05 21:26:31 theblob kernel: amdgpu: Failed to remove queue 0
Dec 05 21:26:33 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: GPU reset begin!
Dec 05 21:26:33 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: recover vram bo from shadow start
Dec 05 21:26:33 theblob kernel: amdgpu 0000:2f:00.0: amdgpu: recover vram bo from shadow done
Dec 05 21:26:33 theblob google-chrome.desktop[11352]: [22281:22281:1205/212633.285036:ERROR:vulkan_swap_chain.cc(404)] vkQueuePresentKHR() failed: -4
Dec 05 21:26:33 theblob google-chrome.desktop[11352]: [22281:22281:1205/212633.285264:ERROR:gpu_service_impl.cc(1105)] Exiting GPU process because some drivers can't recover from errors. GPU process will restart shortly.
Dec 05 21:26:33 theblob google-chrome.desktop[11352]: [11346:11346:1205/212633.301983:ERROR:gpu_process_host.cc(990)] GPU process exited unexpectedly: exit_code=8704only way i can recover is with a full power cycle (off for a few seconds)
Linux theblob 6.2.0-37-generic #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 2 18:01:13 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-====================-===========================-============-====================================================
ii amdgpu-core 1:5.7.50702-1683306.22.04 all Core meta package for unified amdgpu driver.
ii amdgpu-dkms 1:6.2.4.50702-1683306.22.04 all amdgpu driver in DKMS format.
ii amdgpu-dkms-firmware 1:6.2.4.50702-1683306.22.04 all firmware blobs used by amdgpu driver in DKMS format
ii amdgpu-dkms-headers 1:6.2.4.50702-1683306.22.04 all amdgpu driver linux kernel headers
ii amdgpu-install 5.7.50702-1683306.22.04 all AMDGPU driver repository and installer
ii amdgpu-lib 1:5.7.50702-1683306.22.04 amd64 Meta package to install amdgpu userspace components.
ii amdgpu-top 0.2.3 amd64 Tool to displays AMDGPU usage.
sid-cypher
commented
8 months ago I've had this exact issue frequently with ROCm 5.7 on Radeon RX 7900 XTX.
Upgrading to ROCm 6.0 has solved it for me. At least Stable Diffusion with torch-2.3.0+rocm5.7 still works on ROCm 6.0, and without crashes now, no need to restart after a failed GPU reset.
The non-critical GCVM_L2_PROTECTION_FAULT_STATUS errors are still there, but they don't force me to restart after MES failure and a GPU reset attempt.
Currently working:
Linux moon 6.2.0-39-generic #40~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 16 10:53:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
amdgpu-dkms/jammy,now 1:6.3.6.60000-1697589.22.04 all [installed]
amdgpu-install/jammy,now 6.0.60000-1697589.22.04 all [installed]
selenologist
commented
7 months ago I am still seeing this on both ROCm 6.0 and the 6.0.1 packages at https://repo.radeon.com/rocm/apt/6.0.1 / https://repo.radeon.com/amdgpu/apt/6.0.1 on fresh Ubuntu 22.04.
Linux amdsux 6.5.0-15-generic #15~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 12 18:54:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
I realize it does no good for everyone else experiencing this issue to add yet another "yeah, same here" and some venting, so the rest of you may stop reading now. But AMD, you really need to know this is not acceptable.
It's been over a year since the card was released and still the software isn't even stable, let alone all the trouble it takes to get set up vs nVidia. All whilst AMD releases these frankly patronizing announcements about how great and open and "ready" ROCm 6.0 is, "driving an inflection with developers". Erm, yeah... it's making us run away.
Please understand developer sentiment towards AMD is very poor and getting worse, not better. Everyone else like me who is trying to do research but they bought an AMD card because they couldn't afford a nVidia card, is never ever going to buy an AMD GPU again. And we will tell our employers to never ever buy AMD too, and they will listen, because developer time is far more expensive than nVidia's hardware. We should have listened to our friends who told us to just buy nVidia cards, but we won't make this mistake again. This will last years and years. The best you can do now is admit your failures and try to salvage your reputation for CPUs. Because it doesn't matter how good or cheap your GPUs get, we will never risk wasting our time again.
For all this talk about openness, we're all here seeing this issue and does anyone outside AMD even understand what this message means? For that matter, does anyone INSIDE AMD understand what it means? Please explain it to us.
So far as I can tell, "MES" is the MicroEngine Scheduler, and https://docs.kernel.org/gpu/amdgpu/driver-core.html#graphics-and-compute-microcontrollers says it is "unused". Mumblings on the internet suggest it is used now, and well, we're in this GitHub issue together aren't we? Beyond that, things get extremely hazy. The kernel code that actually produces that log message is completely opaque. None of these functions are documented. MES is... a separate microcontroller controlled by a firmware blob? Do I have that right? And what exactly does this mean? https://github.com/torvalds/linux/blob/41bccc98fb7931d63d03f326a746ac4d429c1dd3/drivers/gpu/drm/amd/amdgpu/mes_v11_0.c#L563
Is this AMD's "open software approach"? Hanging binary blobs and sending tarballs to geohot that he's not allowed to distribute?!
PS: I'm still not over the terrible software support for the RX480 last time. I hoped things had changed. Fool me twice, shame on me. But never again. I'm losing my mind over this because I've lost months of productivity... I can't stress enough, my experiment seems to be working whilst I can get it to run. But the reason I cannot publish is because I chose the wrong GPU. I'm going to have to go get a "real job" instead.
mgolub2
commented
6 months ago Also seeing this issue, on ROCm 6.0.2, Ubuntu 22.04 fresh setup when running llama.cpp. Hardware is a AMD 7940HS CPU w/ Radeon 780M GPU. When loading up larger models and offloading more layers to the GPU, most of the time the GPU will reset with the MES panic, or just hang with hundreds of [drm] Skip scheduling IBs! messages. When the GPU hangs like this, modprobe -r amdgpu causes a segfault (!?), and a dmesg report about a kernel bug (!?x2):
[ 6854.988626] ------------[ cut here ]------------
[ 6854.988627] kernel BUG at mm/slub.c:384!
[ 6854.988635] invalid opcode: 0000 [#1] SMP NOPTI
[ 6854.988637] CPU: 1 PID: 193444 Comm: modprobe Tainted: G OE 5.15.0-94-generic #104-Ubuntu
[ 6854.988640] Hardware name: Micro Computer (HK) Tech Limited Venus series/F7BSC, BIOS 1.09 11/20/2023
[ 6854.988641] RIP: 0010:__slab_free+0x1f3/0x370 If an AMD engineer (or anyone else!) could point to a place to investigate, or has any other idea, that would be helpful!
terryrankine
commented
6 months ago 5.15.0-94-generic
Any reason you want to run the older kernel?
A few people are experiencing better results on the newer kernels, and the 6.0.2 drivers.
I got to a running setup (with some minor issues) with the 6.5.x stock one - ended up closing my ticket too. #2689
mgolub2
commented
6 months ago 5.15.0-94-generic
Any reason you want to run the older kernel?
A few people are experiencing better results on the newer kernels, and the 6.0.2 drivers.
I got to a running setup (with some minor issues) with the 6.5.x stock one - ended up closing my ticket too. #2689
I have no idea actually, I just went through a standard ubuntu server 22.04 setup from what I recall (via a netboot, so maybe the installer is missing the HWE option? I did not see it at least), after a quick sudo apt install -y linux-image-generic-hwe-22.04 and reboot, now on kernel 6.5.0-17-generic #17~22.04.1-Ubuntu things don't seem to have improved for me - I still get gpu resets:
│[33693.746423] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
│[33693.746596] amdgpu: failed to remove hardware queue from MES, doorbell=0x1002
│[33693.746606] amdgpu: MES might be in unrecoverable state, issue a GPU reset
│[33693.746614] amdgpu: Failed to evict queue 1
│[33693.746620] amdgpu: Failed to evict process queues
│[33693.746949] amdgpu 0000:c5:00.0: amdgpu: GPU reset begin!
│[33693.917504] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
│[33693.917664] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
│[33693.924953] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
│[33693.925108] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
│[33694.045099] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
│[33694.045218] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
│[33694.052583] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=14
│[33694.052702] [drm:amdgpu_mes_reg_write_reg_wait [amdgpu]] *ERROR* failed to reg_write_reg_wait
│[33694.172644] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3After that happens, the GPU is unusable until a full reboot.
selenologist
commented
6 months ago Okay what the fuck. I've just captured a log with MES-related messages (and some others) from a system that was idle and hadn't even run a real workload yet. I don't know if this bug is related (because I don't have a clue what's on the other side of your blob wall or what much of this opaque code does) but there are MES messages, so, well, here we go.
[31317.644104] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31317.644111] amdgpu 0000:0d:00.0: amdgpu: Failed to export SMU metrics table!
[31317.644112] amdgpu 0000:0d:00.0: amdgpu: Failed to get current clock freq!
[31321.920024] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31321.920028] amdgpu 0000:0d:00.0: amdgpu: Failed to export SMU metrics table!
[31326.191767] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31326.191772] amdgpu 0000:0d:00.0: amdgpu: Failed to export SMU metrics table!
[31327.729436] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx_0.0.0 timeout, signaled seq=613206, emitted seq=613208
[31327.729995] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process plasmashell pid 3182 thread plasmashel:cs0 pid 3224
[31327.730545] amdgpu 0000:0d:00.0: amdgpu: GPU reset begin!
[31328.743414] amdgpu 0000:0d:00.0: amdgpu: IP block:gfx_v11_0 is hung!
[31329.767382] [drm:sdma_v6_0_ring_test_ib [amdgpu]] *ERROR* amdgpu: IB test timed out
[31329.767916] amdgpu 0000:0d:00.0: amdgpu: IP block:sdma_v6_0 is hung!
[31329.768161] [drm] kiq ring mec 3 pipe 1 q 0
[31330.008807] amdgpu 0000:0d:00.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring mes_kiq_3.1.0 test failed (-110)
[31330.009039] [drm:amdgpu_gfx_enable_kcq [amdgpu]] *ERROR* KCQ enable failed
[31330.009290] amdgpu 0000:0d:00.0: amdgpu: soft reset failed, will fallback to full reset!
[31330.479483] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31330.479487] amdgpu 0000:0d:00.0: amdgpu: Failed to export SMU metrics table!
[31334.746500] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31334.746505] amdgpu 0000:0d:00.0: amdgpu: Failed to disable gfxoff!
[31339.013104] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31339.013107] amdgpu 0000:0d:00.0: amdgpu: [SetDfCstate] failed!
[31339.013109] amdgpu 0000:0d:00.0: amdgpu: Failed to disallow df cstate
[31341.566559] [drm] psp gfx command INVOKE_CMD(0x3) failed and response status is (0x0)
[31345.514564] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31345.514834] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31345.636256] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31345.636512] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31345.757963] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31345.758218] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31345.879631] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31345.879887] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31346.001297] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31346.001552] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31346.122973] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31346.123230] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31346.244645] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31346.244900] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31346.366313] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31346.366568] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31346.487977] [drm:mes_v11_0_submit_pkt_and_poll_completion.constprop.0 [amdgpu]] *ERROR* MES failed to response msg=3
[31346.488232] [drm:amdgpu_mes_unmap_legacy_queue [amdgpu]] *ERROR* failed to unmap legacy queue
[31348.539759] [drm] psp gfx command UNLOAD_TA(0x2) failed and response status is (0x0)
[31348.539764] [drm:psp_suspend [amdgpu]] *ERROR* Failed to terminate ras ta
[31348.540218] [drm:amdgpu_device_ip_suspend_phase2 [amdgpu]] *ERROR* suspend of IP block <psp> failed -22
[31348.541779] amdgpu 0000:0d:00.0: amdgpu: MODE1 reset
[31348.541783] amdgpu 0000:0d:00.0: amdgpu: GPU mode1 reset
[31348.541843] amdgpu 0000:0d:00.0: amdgpu: GPU smu mode1 reset
[31352.825254] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005
[31352.825259] amdgpu 0000:0d:00.0: amdgpu: GPU mode1 reset failed
[31352.825334] amdgpu 0000:0d:00.0: amdgpu: ASIC reset failed with error, -62 for drm dev, 0000:0d:00.0
[31357.080573] amdgpu 0000:0d:00.0: amdgpu: GPU reset succeeded, trying to resume
[31357.080830] [drm] PCIE GART of 512M enabled (table at 0x0000008000800000).
[31357.080897] [drm] VRAM is lost due to GPU reset!
[31357.080898] [drm] PSP is resuming...
[31357.320464] [drm:psp_hw_start [amdgpu]] *ERROR* PSP create ring failed!
[31357.320721] [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed
[31357.320968] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -62
[31357.321238] [drm] Skip scheduling IBs!
[31357.321257] [drm] Skip scheduling IBs!I'm thinking it's time to RMA my card. This is unfair to the card manufacturer and retailer who have to deal with this, only reason I've been putting it off. But come on, really, what is this?
Full details:
I returned to my Kubuntu install that I was trying to do real work on, but after cold booting the system and logging in, I ended up just going to bed. Aside from displaying a KDE plasma desktop on Wayland, the GPU didn't do anything taxing. I woke up to find my screen off and system unresponsive, so I logged in over SSH from another computer and retrieved this log from dmesg. Kernel is 6.5.0-17-generic.
So, wait... AMD cards aren't even stable handling a basic plasmashell desktop? Really?
Also I want to take this opportunity to clarify my position to AMD's engineers. I'm not mad at you. You guys probably hate this too. I think we're on the same team. I'm furious at the company and the situation though, because I feel lied to, scammed even. My messages here are not intended to be hostile, however, and I want to indicate that I am, you know, capable of calm discussion. I'm not going to just jump on anyone who tries to help.
PS: if recent kernels help, with patches not backported into the fresh-install Ubuntu kernel the AMD documentation implicitly assumes, then said documentation should include instructions on installing an updated kernel. Thanks for bringing that up terryrankine.
mgolub2
commented
6 months ago PS: if recent kernels help, with patches not backported into the fresh-install Ubuntu kernel the AMD documentation implicitly assumes, then said documentation should include instructions on installing an updated kernel. Thanks for bringing that up terryrankine.
Sadly it did not help me :( High VRAM workloads still cause the MES unrecoverable error, every time, which requires a reboot to solve.
selenologist
commented
6 months ago Sadly it did not help me :( High VRAM workloads still cause the MES unrecoverable error, every time, which requires a reboot to solve.
That is also the case for me - hey, it's weird how using more VRAM makes it more likely, isn't? Maybe some allocator race? ... really though, AMD should be paying me even to speculate about what's going wrong on the proprietary side.
Also I said before that I thought it wasn't helpful for me to add a "same here" but I've realized, anyone having this issue should absolutely post about it here. Let 'em see us.
From statements like "We are hoping that this will improve your perception of AMD products and this will be reflected in your public messaging", we know that public perception is what the company cares about, not whether people are having an issue. Let your public messaging reflect your actual perception.
terryrankine
commented
6 months ago and just checking - you are on new kernels with the 6.0.2 binaries or your code compiled against the 6.0.2 libs? (yes - i have had to compile pytorch against 6.0.2).
mainly because i finally got there - and closed my ticket.... #2689
Im not saying it wasnt easy or simple but I can hit 20-22Gb consistently somehow now.... and Im over like 3 days of uptime too... which is a first since november last year...
terryr@theblob:~$ uname -a
Linux theblob 6.5.0-18-generic #18~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 11:40:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linuxterryr@theblob:~$ hipconfig
HIP version : 6.0.32831-204d35d16
== hipconfig
HIP_PATH : /opt/rocm-6.0.2
ROCM_PATH : /opt/rocm-6.0.2
HIP_COMPILER : clang
HIP_PLATFORM : amd
HIP_RUNTIME : rocclr
CPP_CONFIG : -D__HIP_PLATFORM_HCC__= -D__HIP_PLATFORM_AMD__= -I/opt/rocm-6.0.2/include -I/opt/rocm-6.0.2/lib/llvm/lib/clang/17.0.0and here is the worst ever way to show what ubuntu libs got me there....
terryr@theblob:~$ dpkg -l | grep -E "rocm|hip|AMD "
ii amd-smi-lib 23.4.2.60002-115~22.04 amd64 AMD System Management libraries
ii amd64-microcode 3.20191218.1ubuntu2.2 amd64 Processor microcode firmware for AMD CPUs
ii composablekernel-dev 1.1.0.60002-115~22.04 amd64 High Performance Composable Kernel for AMD GPUs
ii hip-dev 6.0.32831.60002-115~22.04 amd64 HIP:Heterogenous-computing Interface for Portability
ii hip-doc 6.0.32831.60002-115~22.04 amd64 HIP:Heterogenous-computing Interface for Portability
ii hip-runtime-amd 6.0.32831.60002-115~22.04 amd64 HIP:Heterogenous-computing Interface for Portability
ii hip-samples 6.0.32831.60002-115~22.04 amd64 HIP: Heterogenous-computing Interface for Portability [HIP SAMPLES]
ii hipblas 2.0.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library
ii hipblas-dev 2.0.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library
ii hipblaslt 0.6.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library
ii hipblaslt-dev 0.6.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library
ii hipcc 1.0.0.60002-115~22.04 amd64 HIP Compiler Driver
ii hipcub-dev 3.0.0.60002-115~22.04 amd64 hipCUB (rocPRIM backend)
ii hipfft 1.0.13.60002-115~22.04 amd64 ROCm FFT marshalling library
ii hipfft-dev 1.0.13.60002-115~22.04 amd64 ROCm FFT marshalling library
ii hipfort-dev 0.4.0.60002-115~22.04 amd64 Fortran Interface For GPU Kernel Libraries
ii hipify-clang 17.0.0.60002-115~22.04 amd64 Hipify CUDA source
ii hiprand 2.10.16.60002-115~22.04 amd64 Radeon Open Compute RAND library
ii hiprand-dev 2.10.16.60002-115~22.04 amd64 Radeon Open Compute RAND library
ii hipsolver 2.0.0.60002-115~22.04 amd64 Radeon Open Compute LAPACK marshalling library
ii hipsolver-dev 2.0.0.60002-115~22.04 amd64 Radeon Open Compute LAPACK marshalling library
ii hipsparse 3.0.0.60002-115~22.04 amd64 ROCm SPARSE library
ii hipsparse-dev 3.0.0.60002-115~22.04 amd64 ROCm SPARSE library
ii hiptensor 1.1.0.60002-115~22.04 amd64 Adaptation library of tensor contraction with composable_kernel backend
ii hiptensor-dev 1.1.0.60002-115~22.04 amd64 Adaptation library of tensor contraction with composable_kernel backend
ii hsa-amd-aqlprofile 1.0.0.60002.60002-115~22.04 amd64 AQLPROFILE library for AMD HSA runtime API extension support
ii hsa-rocr 1.12.0.60002-115~22.04 amd64 AMD Heterogeneous System Architecture HSA - Linux HSA Runtime for Boltzmann (ROCm) platforms
ii hsa-rocr-dev 1.12.0.60002-115~22.04 amd64 AMD Heterogeneous System Architecture HSA development package.
ii libflashrom1:amd64 1.2-5build1 amd64 Identify, read, write, erase, and verify BIOS/ROM/flash chips - library
ii libresid-builder0c2a 2.1.1-15ubuntu2 amd64 SID chip emulation class based on resid
ii miopen-hip 3.00.0.60002-115~22.04 amd64 AMD's DNN Library
ii miopen-hip-dev 3.00.0.60002-115~22.04 amd64 AMD's DNN Library
ii mivisionx 2.5.0.60002-115~22.04 amd64 AMD MIVisionX is a comprehensive Computer Vision and ML Inference Toolkit
ii rocblas 4.0.0.60002-115~22.04 amd64 rocBLAS is AMD's library for BLAS on ROCm. It is implemented in HIP and optimized for AMD GPUs.
ii rocblas-dev 4.0.0.60002-115~22.04 amd64 rocBLAS is AMD's library for BLAS on ROCm. It is implemented in HIP and optimized for AMD GPUs.
ii rocm 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) software stack meta package
ii rocm-clang-ocl 0.5.0.60002-115~22.04 amd64 OpenCL compilation with clang compiler.
ii rocm-cmake 0.11.0.60002-115~22.04 amd64 rocm-cmake built using CMake
ii rocm-core 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-dbgapi 0.71.0.60002-115~22.04 amd64 Library to provide AMD GPU debugger API
ii rocm-debug-agent 2.0.3.60002-115~22.04 amd64 Radeon Open Compute Debug Agent (ROCdebug-agent)
ii rocm-dev 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-developer-tools 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-device-libs 1.0.0.60002-115~22.04 amd64 Radeon Open Compute - device libraries
ii rocm-gdb 13.2.60002-115~22.04 amd64 ROCgdb
ii rocm-hip-libraries 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-hip-runtime 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-hip-runtime-dev 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-hip-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-language-runtime 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-libs 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-llvm 17.0.0.24012.60002-115~22.04 amd64 ROCm compiler
ii rocm-ml-libraries 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-ml-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-ocl-icd 2.0.0.60002-115~22.04 amd64 clr built using CMake
ii rocm-opencl 2.0.0.60002-115~22.04 amd64 clr built using CMake
ii rocm-opencl-dev 2.0.0.60002-115~22.04 amd64 clr built using CMake
ii rocm-opencl-runtime 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-opencl-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocm-openmp-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) OpenMP Software development Kit.
ii rocm-smi-lib 6.0.0.60002-115~22.04 amd64 AMD System Management libraries
ii rocm-utils 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack
ii rocminfo 1.0.0.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime rocminfo tool
ii rocprofiler 2.0.60002.60002-115~22.04 amd64 ROCPROFILER library for AMD HSA runtime API extension support
ii rocprofiler-dev 2.0.60002.60002-115~22.04 amd64 ROCPROFILER library for AMD HSA runtime API extension support
ii rocprofiler-plugins 2.0.60002.60002-115~22.04 amd64 ROCPROFILER library for AMD HSA runtime API extension support
ii rocsolver 3.24.0.60002-115~22.04 amd64 AMD ROCm SOLVER library
ii rocsolver-dev 3.24.0.60002-115~22.04 amd64 AMD ROCm SOLVER library
ii roctracer 4.1.60002.60002-115~22.04 amd64 AMD ROCTRACER library
ii roctracer-dev 4.1.60002.60002-115~22.04 amd64 AMD ROCTRACER libraryand then made pytorch find the folder and run the special pytorch convert from cuda to hip and then build, and then build and install, and then build pytorchvision and audio after.... so much faf.....
But - I run now....
mergmann
commented
6 months ago On my arch installation I have run into this issue multiple times with rocm 5.6-6.0. I can't even open blender or libreoffice with hardware acceleration enabled. llama.cpp (with CLBlast) crashes even without offloading layers to GPU Especially OpenCL workloads seem to be problematic. Running a very simple kernel arr[get_global_id(0)] += num; triggers the page fault error, but that seems to not be a problem. Looping over that code 1024*1024*1024 times causes the driver to timeout and reset recoverably. I even tried allocating a 16GB buffer, which worked fine. I wasn't able to reproduce the MES error. However starting llama.cpp caused it again. On ubuntu pytorch mostly works, but blender and llama.cpp still cause unrecoverable gpu crashes. I hope this gets fixed soon as I wasn't able to use my card for anything else than graphics. I never really had problems with vulkan or OpenGL, it's just ROCm that appears to be broken.
mgolub2
commented
6 months ago and just checking - you are on new kernels with the 6.0.2 binaries or your code compiled against the 6.0.2 libs? (yes - i have had to compile pytorch against 6.0.2).
mainly because i finally got there - and closed my ticket.... #2689
Im not saying it wasnt easy or simple but I can hit 20-22Gb consistently somehow now.... and Im over like 3 days of uptime too... which is a first since november last year...
terryr@theblob:~$ uname -a Linux theblob 6.5.0-18-generic #18~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 11:40:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linuxterryr@theblob:~$ hipconfig HIP version : 6.0.32831-204d35d16 == hipconfig HIP_PATH : /opt/rocm-6.0.2 ROCM_PATH : /opt/rocm-6.0.2 HIP_COMPILER : clang HIP_PLATFORM : amd HIP_RUNTIME : rocclr CPP_CONFIG : -D__HIP_PLATFORM_HCC__= -D__HIP_PLATFORM_AMD__= -I/opt/rocm-6.0.2/include -I/opt/rocm-6.0.2/lib/llvm/lib/clang/17.0.0and here is the worst ever way to show what ubuntu libs got me there....
terryr@theblob:~$ dpkg -l | grep -E "rocm|hip|AMD " ii amd-smi-lib 23.4.2.60002-115~22.04 amd64 AMD System Management libraries ii amd64-microcode 3.20191218.1ubuntu2.2 amd64 Processor microcode firmware for AMD CPUs ii composablekernel-dev 1.1.0.60002-115~22.04 amd64 High Performance Composable Kernel for AMD GPUs ii hip-dev 6.0.32831.60002-115~22.04 amd64 HIP:Heterogenous-computing Interface for Portability ii hip-doc 6.0.32831.60002-115~22.04 amd64 HIP:Heterogenous-computing Interface for Portability ii hip-runtime-amd 6.0.32831.60002-115~22.04 amd64 HIP:Heterogenous-computing Interface for Portability ii hip-samples 6.0.32831.60002-115~22.04 amd64 HIP: Heterogenous-computing Interface for Portability [HIP SAMPLES] ii hipblas 2.0.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library ii hipblas-dev 2.0.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library ii hipblaslt 0.6.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library ii hipblaslt-dev 0.6.0.60002-115~22.04 amd64 Radeon Open Compute BLAS marshalling library ii hipcc 1.0.0.60002-115~22.04 amd64 HIP Compiler Driver ii hipcub-dev 3.0.0.60002-115~22.04 amd64 hipCUB (rocPRIM backend) ii hipfft 1.0.13.60002-115~22.04 amd64 ROCm FFT marshalling library ii hipfft-dev 1.0.13.60002-115~22.04 amd64 ROCm FFT marshalling library ii hipfort-dev 0.4.0.60002-115~22.04 amd64 Fortran Interface For GPU Kernel Libraries ii hipify-clang 17.0.0.60002-115~22.04 amd64 Hipify CUDA source ii hiprand 2.10.16.60002-115~22.04 amd64 Radeon Open Compute RAND library ii hiprand-dev 2.10.16.60002-115~22.04 amd64 Radeon Open Compute RAND library ii hipsolver 2.0.0.60002-115~22.04 amd64 Radeon Open Compute LAPACK marshalling library ii hipsolver-dev 2.0.0.60002-115~22.04 amd64 Radeon Open Compute LAPACK marshalling library ii hipsparse 3.0.0.60002-115~22.04 amd64 ROCm SPARSE library ii hipsparse-dev 3.0.0.60002-115~22.04 amd64 ROCm SPARSE library ii hiptensor 1.1.0.60002-115~22.04 amd64 Adaptation library of tensor contraction with composable_kernel backend ii hiptensor-dev 1.1.0.60002-115~22.04 amd64 Adaptation library of tensor contraction with composable_kernel backend ii hsa-amd-aqlprofile 1.0.0.60002.60002-115~22.04 amd64 AQLPROFILE library for AMD HSA runtime API extension support ii hsa-rocr 1.12.0.60002-115~22.04 amd64 AMD Heterogeneous System Architecture HSA - Linux HSA Runtime for Boltzmann (ROCm) platforms ii hsa-rocr-dev 1.12.0.60002-115~22.04 amd64 AMD Heterogeneous System Architecture HSA development package. ii libflashrom1:amd64 1.2-5build1 amd64 Identify, read, write, erase, and verify BIOS/ROM/flash chips - library ii libresid-builder0c2a 2.1.1-15ubuntu2 amd64 SID chip emulation class based on resid ii miopen-hip 3.00.0.60002-115~22.04 amd64 AMD's DNN Library ii miopen-hip-dev 3.00.0.60002-115~22.04 amd64 AMD's DNN Library ii mivisionx 2.5.0.60002-115~22.04 amd64 AMD MIVisionX is a comprehensive Computer Vision and ML Inference Toolkit ii rocblas 4.0.0.60002-115~22.04 amd64 rocBLAS is AMD's library for BLAS on ROCm. It is implemented in HIP and optimized for AMD GPUs. ii rocblas-dev 4.0.0.60002-115~22.04 amd64 rocBLAS is AMD's library for BLAS on ROCm. It is implemented in HIP and optimized for AMD GPUs. ii rocm 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) software stack meta package ii rocm-clang-ocl 0.5.0.60002-115~22.04 amd64 OpenCL compilation with clang compiler. ii rocm-cmake 0.11.0.60002-115~22.04 amd64 rocm-cmake built using CMake ii rocm-core 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-dbgapi 0.71.0.60002-115~22.04 amd64 Library to provide AMD GPU debugger API ii rocm-debug-agent 2.0.3.60002-115~22.04 amd64 Radeon Open Compute Debug Agent (ROCdebug-agent) ii rocm-dev 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-developer-tools 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-device-libs 1.0.0.60002-115~22.04 amd64 Radeon Open Compute - device libraries ii rocm-gdb 13.2.60002-115~22.04 amd64 ROCgdb ii rocm-hip-libraries 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-hip-runtime 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-hip-runtime-dev 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-hip-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-language-runtime 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-libs 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-llvm 17.0.0.24012.60002-115~22.04 amd64 ROCm compiler ii rocm-ml-libraries 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-ml-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-ocl-icd 2.0.0.60002-115~22.04 amd64 clr built using CMake ii rocm-opencl 2.0.0.60002-115~22.04 amd64 clr built using CMake ii rocm-opencl-dev 2.0.0.60002-115~22.04 amd64 clr built using CMake ii rocm-opencl-runtime 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-opencl-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocm-openmp-sdk 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) OpenMP Software development Kit. ii rocm-smi-lib 6.0.0.60002-115~22.04 amd64 AMD System Management libraries ii rocm-utils 6.0.2.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack ii rocminfo 1.0.0.60002-115~22.04 amd64 Radeon Open Compute (ROCm) Runtime rocminfo tool ii rocprofiler 2.0.60002.60002-115~22.04 amd64 ROCPROFILER library for AMD HSA runtime API extension support ii rocprofiler-dev 2.0.60002.60002-115~22.04 amd64 ROCPROFILER library for AMD HSA runtime API extension support ii rocprofiler-plugins 2.0.60002.60002-115~22.04 amd64 ROCPROFILER library for AMD HSA runtime API extension support ii rocsolver 3.24.0.60002-115~22.04 amd64 AMD ROCm SOLVER library ii rocsolver-dev 3.24.0.60002-115~22.04 amd64 AMD ROCm SOLVER library ii roctracer 4.1.60002.60002-115~22.04 amd64 AMD ROCTRACER library ii roctracer-dev 4.1.60002.60002-115~22.04 amd64 AMD ROCTRACER libraryand then made pytorch find the folder and run the special pytorch convert from cuda to hip and then build, and then build and install, and then build pytorchvision and audio after.... so much faf.....
But - I run now....
Yes, I made sure to reinstall rocm 6.0.2, and do a clean build of llama.cpp - No change for me sadly. I might try it again to see if I missed something, always possible.
Out of curiosity, what mode is you apu in for VRAM? auto or do you have it set to dedicated mode?
selenologist
commented
6 months ago I might have a workaround that maybe helps a little, or it might just be placebo / luck so far. There are potential drawbacks, though.
I noticed that sometimes, but not always, there would be IOMMU fault messages prior to the ones from the amdgpu driver itself. So I thought, hey why not, let's try disabling IOMMU in BIOS. And so far, I've been able to run things ... I don't trust it, obviously even if this works it's not a solution, but I thought I'd share. Those faults were obviously happening for a reason and this seems to just be putting a blindfold on.
Here's a chat with Gemini about the consequences of disabling IOMMU https://g.co/gemini/share/5d21ada997c1 , please weigh it up yourself before trying it. I don't understand IOMMU properly myself.
PS: I captured this earlier after booting my system and leaving it idle, not even logging in yet:
[ 555.894238] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005(and then the usual circus follows - this is just the start for demonstration) So not even 10 minutes, under no load, with nothing loaded except a Kubuntu login screen. Something is very wrong.
mergmann
commented
6 months ago Yesterday I tried running some AI workloads with ROCm 6.0.0 and PyTorch ROCm 6.0 on my main system, but already very simple tensor operations with pytorch crashed the driver with the MES failed to response messages. I then booted into ubuntu (the same version of ROCm), retried the same tensor operations there and it worked mostly fine. I then upgraded to ROCm on ubuntu to the newest version and it still worked. Today I booted into my arch installation again, which still has ROCm 6.0.0, but even running heavy workloads like Stable Diffusion and llama.cpp work fine, even at 18/20GB VRAM. My best guess is that upgrading ROCm on ubuntu also installed a firmware update for the GPU, but I'm not certain that the firmware was the root cause.
Anyways, I'll watch how well ROCm performs over the next few days as there are still harmless page faults appearing out of nowhere.
briansp2020
commented
6 months ago @MattisBergmann Did you try ROCm 6.0.2? I think it fixed some driver issues and it was much more stable for me.
FeepingCreature
commented
5 months ago I'm getting "MES failed to response" with ROCM 6.0.2, freshly updated.
Disty0
commented
5 months ago Using an RX 7900 XTX with no monitor attached, training a diffusion model or just running inference. (Getting monitor output from another GPU.)
Still seeing the same MES hangs with Arch Linux 6.8.1-arch1-1 / ROCm 6.0.2 / PyTorch 2.4.0.dev20240323+rocm6.0 This MES hang causes other unrelated Python programs to core dump too and GPU keeps pulling 120W~ of power when it hangs.
Only way to recover from the hang is running rocm-smi --gpureset -id 0 if you have no monitor attached to the AMD GPU.
If you have a monitor attached, only way to recover is hard rebooting.
Adding amdgpu.mes=0 amdgpu.mes_kiq=1 to the grub config doesn't seem to be doing anything either.
Log on PyTorch when MES hangs:
HW Exception by GPU node-1 (Agent handle: 0x62e2bc7e39b0) reason :GPU Hang
HW Exception by GPU node-1 (Agent handle: 0x5fb477fb5bd0) reason :GPU HangJournalctl in reverse order:
FeepingCreature
commented
5 months ago Yeah it keeps happening here with 6.0.2 as well.
nonetrix
commented
5 months ago I might have a workaround that maybe helps a little, or it might just be placebo / luck so far. There are potential drawbacks, though.
I noticed that sometimes, but not always, there would be IOMMU fault messages prior to the ones from the amdgpu driver itself. So I thought, hey why not, let's try disabling IOMMU in BIOS. And so far, I've been able to run things ... I don't trust it, obviously even if this works it's not a solution, but I thought I'd share. Those faults were obviously happening for a reason and this seems to just be putting a blindfold on.
Here's a chat with Gemini about the consequences of disabling IOMMU https://g.co/gemini/share/5d21ada997c1 , please weigh it up yourself before trying it. I don't understand IOMMU properly myself.
PS: I captured this earlier after booting my system and leaving it idle, not even logging in yet:
[ 555.894238] amdgpu 0000:0d:00.0: amdgpu: SMU: I'm not done with your previous command: SMN_C2PMSG_66:0x00000012 SMN_C2PMSG_82:0x00000005(and then the usual circus follows - this is just the start for demonstration) So not even 10 minutes, under no load, with nothing loaded except a Kubuntu login screen. Something is very wrong.
Hey, I got the same issue that you are having somewhat with my RX 6800 same error you mentioned and exact same command it's trying to sent(SMN_C2PMSG_66 and SMN_C2PMSG_82) https://gitlab.freedesktop.org/drm/amd/-/issues/3220 but on my issue they mentioned that a fix is there but hasn't been released maybe relevant? It's not available for testing though :/
Something tells me this is a race condition perhaps lol
romeo4934
commented
5 months ago Same issue. Any help?
yshui
commented
5 months ago from my own observations, the MES seems to destabilize as the usable VRAM becomes low. I could be very wrong about this, but I tried limiting the amount of usable VRAM with module parameters (vramlimit, vis_vramlimit), and that does seem to help.
Interested to see if this works for others.
FeepingCreature
commented
4 months ago Okay, I don't want to hype anybody up unduly, but I haven't seen a MES hang in two days since I installed the week-old AMD firmware update. Can anyone repro on that?
edit: Nope, still happens.
nonetrix
commented
4 months ago Will see if it fixes my unrelated issue as well, apparently they where working on fixing mine already and had a fixed testing unreleased firmware as of month ago. Hoping AMD gets their act together in general which is why I mention, not sure if my distro is running it though how could I even tell exactly?
FeepingCreature
commented
4 months ago You can just clone the repo, git clone git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git --depth 1, then sudo cp linux-firmware/amdgpu/* /lib/firmware/updates/amdgpu/ and reboot. Thanks @saffroy at https://gitlab.freedesktop.org/drm/amd/-/issues/3175 for pointing this out; note that this firmware is relatively untested, hot off the presses, and also your system will overwrite this when it updates anyway, I take no responsibility for damages etc. But also I've been running it for two days and my PC hasn't caught fire yet.
If you've installed it correctly, you should note a numeric change in grep MES /sys/kernel/debug/dri/{0,1}/amdgpu_firmware_info. Mine is at MES 0x0000004b, MES_KIQ 0x00000075.
edit: But it crashes anyway.
nonetrix
commented
4 months ago So far not having any crashes with my distros latest, still having the cursor stutters bug perhaps unrelated and if so that's just mildly annoying and can live with that. Don't want to jinx it though so going to let to generate idle, if it crashes I guess I'll look into updating my firmware. Again, I suspect my issue is unrelated but seems others in thread where having mine
Edit: got a crash, seems more stable at least though
FeepingCreature
commented
4 months ago Nevermind everyone, sorry for the premature excitement. Two days later, there it goes again.
amdgpu 0000:03:00.0: amdgpu: MES might be in unrecoverable state, issue a GPU resetOf course, it had to wait for me to post this to crash...
nonetrix
commented
4 months ago
grep MES /sys/kernel/debug/dri/{0,1}/amdgpu_firmware_info
Huh mine seems to say
/sys/kernel/debug/dri/1/amdgpu_firmware_info:MES_KIQ feature version: 0, firmware version: 0x00000000
/sys/kernel/debug/dri/1/amdgpu_firmware_info:MES feature version: 0, firmware version: 0x00000000Could it be that my card is too old or something? Pretty sure it was applied though, running Ubuntu just for testing, also going to try to run ROCm 6.1 which might have some fixes for issue I am having as well as others. I did have to change the command to use /lib/firmware/amdgpu/ instead though. Here is the full output of that
VCE feature version: 0, firmware version: 0x00000000
UVD feature version: 0, firmware version: 0x00000000
MC feature version: 0, firmware version: 0x00000000
ME feature version: 44, firmware version: 0x00000040
PFP feature version: 44, firmware version: 0x00000062
CE feature version: 44, firmware version: 0x00000025
RLC feature version: 1, firmware version: 0x00000060
RLC SRLC feature version: 0, firmware version: 0x00000000
RLC SRLG feature version: 0, firmware version: 0x00000000
RLC SRLS feature version: 0, firmware version: 0x00000000
RLCP feature version: 0, firmware version: 0x00000000
RLCV feature version: 0, firmware version: 0x00000000
MEC feature version: 44, firmware version: 0x00000076
MEC2 feature version: 44, firmware version: 0x00000076
IMU feature version: 0, firmware version: 0x00000000
SOS feature version: 0, firmware version: 0x00210e64
ASD feature version: 553648338, firmware version: 0x210000d2
TA XGMI feature version: 0x00000000, firmware version: 0x2000000f
TA RAS feature version: 0x00000000, firmware version: 0x1b00013e
TA HDCP feature version: 0x00000000, firmware version: 0x1700003e
TA DTM feature version: 0x00000000, firmware version: 0x12000016
TA RAP feature version: 0x00000000, firmware version: 0x07000016
TA SECUREDISPLAY feature version: 0x00000000, firmware version: 0x00000000
SMC feature version: 0, program: 0, firmware version: 0x003a5900 (58.89.0)
SDMA0 feature version: 52, firmware version: 0x00000053
SDMA1 feature version: 52, firmware version: 0x00000053
SDMA2 feature version: 52, firmware version: 0x00000053
SDMA3 feature version: 52, firmware version: 0x00000053
VCN feature version: 0, firmware version: 0x0311e008
DMCU feature version: 0, firmware version: 0x00000000
DMCUB feature version: 0, firmware version: 0x02020020
TOC feature version: 0, firmware version: 0x00000000
MES_KIQ feature version: 0, firmware version: 0x00000000
MES feature version: 0, firmware version: 0x00000000
VPE feature version: 0, firmware version: 0x00000000
VBIOS version: 113-1MS21XL203W_210810I manually reinstalled the linux-firmware package and nothing changed, so I think I am doing it wrong
Disty0
commented
4 months ago This MES crash tends to segfault other unrelated programs as well.
This one in particular was interesting. Entire system locked up shortly after this screenshot: (I use an Intel dGPU for video out.)
OS: Arch Linux 6.8.4 CPU: AMD Ryzen 7 5800X3D GPU0: AMD RX 7900 XTX Sapphire Pulse (No video output, runs only PyTorch) GPU1: Intel ARC A770 LE (Used for video output and genral use)
ROCm: 6.0.2 PyTorch: 2.3.0+rocm6.0
Journalctl in reverse order:
galister
commented
3 months ago I've switched my desktop to use the iGPU (7800X3D) and it seems like the driver doesn't hang anymore, as long as PyTorch is the only thing using the XTX.
Edit: I also forced the iGPU to be the primary graphics device via BIOS and plugged all displays into mobo.
ROCm 6.1 in an Ubuntu 22.04 distrobox under Gentoo with torch-2.4.0.dev20240518+rocm6.1
Edit: 2 days later, no sign of issue
hartmark
commented
2 weeks ago I still get this error. Especially with Stable Diffusion generation with FLUX. Is there any update on this issue?
I have this hardware: AMD Ryzen 9 5900X 12-Core 32 GB RAM AMD 7800 XT 16 GB VRAM
I have gotten it to hang consistently 6 times in row now.
I have created a docker image for ComfyUI.
journal: journalctl.txt
Disty0
commented
2 weeks ago Use Linux 6.10, I am using it sine 6.10-rc4 and no crash so far.
hartmark
commented
2 weeks ago Use Linux 6.10, I am using it sine 6.10-rc4 and no crash so far.
Arch linux is at 6.10 already:
% uname -a
Linux bernard 6.10.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Thu, 15 Aug 2024 00:25:30 +0000 x86_64 GNU/Linux
Triggered by running https://github.com/RadeonOpenCompute/rocm_bandwidth_test in a loop while running https://github.com/ROCm-Developer-Tools/HIP-Examples/tree/master/gpu-burn in a loop.
1x 7900XTX ASROCK ROMED8-2T EPYC 7662 Ubuntu 22.04, Kernel 6.2.14-060214-generic, ROCm 5.5
sudo cat /sys/kernel/debug/dri/1/amdgpu_gpu_recoverwill recover the GPU.