search

ROCm / llvm-project

This is the AMD-maintained fork of the LLVM git repository. This repository accepts pull requests and issues related to AMD fork-specific topics (amd/*). For all other issues/PRs, please submit upstream at https://github.com/llvm/llvm-project.
Other
99 stars 52 forks source link

[Issue]: Atomic optimizer reorder causes memory access fault in Blender #58

Open GZGavinZhao opened 3 months ago

GZGavinZhao commented 3 months ago

Problem Description

Following https://projects.blender.org/blender/blender/issues/112084, I've bisected the rocm-6.0.x branch and found that commit 30a3adf50e2d49dfc97c1b614d9b93638eba672d caused any Blender render (using HIP, of course) to crash with message along the lines of "Memory access fault by GPU node-1 (Agent handle: 0x7f1db8337e00) on address 0x7f1bf177e000. Reason: Page not present or supervisor privilege."

Operating System

Solus 4.5 Resilience

CPU

AMD Ryzen 7 5800H with Radeon Graphics

GPU

AMD Instinct MI250, AMD Radeon VII

ROCm Version

ROCm 6.0.0

ROCm Component

llvm-project

Steps to Reproduce

  1. Build this project at the commit mentioned.

  2. Download the Blender 4.1 release binaries: curl -O https://download.blender.org/release/Blender4.1/blender-4.1.0-linux-x64.tar.xz, tar xf blender-4.1.0-linux-x64.tar.xz. You should now have a folder blender-4.1.0-linux-x64.

  3. Clone Blender. Just cloning the v4.1.0 tag is enough: git clone https://projects.blender.org/blender/blender.git --depth 1 --branch v4.1.0.

  4. In the Blender repo, compile the HIP fatbin used to run Blender render: hipcc --offload-arch=$arch --genco intern/cycles/kernel/device/hip/kernel.cpp -D CCL_NAMESPACE_BEGIN= -D CCL_NAMESPACE_END= -D HIPCC -I intern/cycles/kernel/.. -I intern/cycles/kernel/device/hip -ffast-math -o kernel_$arch.fatbin. Adjust HIP_ROCCLR_HOME, HIP_CLANG_PATH as necessary to point to the Clang you just compiled. Replace $arch with the GPU architecture to run on, e.g. gfx900 or gfx1030. Don't add extra attributes like :xnack-.

    If you want to run on multiple architectures, repeat step 4 and 5 for each architecture.

  5. Put this file into blender-4.1.0-linux-x64/4.1/scripts/addons/cycles/lib/kernel_$arch.fatbin.

  6. Get the BMW27 Blender demo file. curl -O https://download.blender.org/demo/test/BMW27.blend.zip, unzip BMW27.blend.zip. You should have a file BMW27.blend.

  7. Now run Blender render. blender-4.1.0-linux-x64/blender -b <path-to-BMW27.blend> -f 0 -- --cycles-device HIP. By default it runs on GPU with device ID 0, so adjust HIP_VISIBLE_DEVICES as necessary to run on the desired GPU.

    You should almost immediately see Blender crash with an error message similar to "Memory access fault by GPU node-1 (Agent handle: 0x7f1db8337e00) on address 0x7f1bf177e000. Reason: Page not present or supervisor privilege."

  8. Now, build LLVM at 1 commit prior, e.g. git switch --detach 30a3adf50e2d49dfc97c1b614d9b93638eba672d~1. Repeat step 4-7, and Blender should render normally.

All of this is on ROCm 6.0.0. If you get a hang instead of a crash when running Blender (likely your on an APU), Ctrl+C and run again with environment variable HSA_ENABLE_SMDA=0.

(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support

rocminfo --support ``` ROCk module is loaded ===================== HSA System Attributes ===================== Runtime Version: 1.1 System Timestamp Freq.: 1000.000000MHz Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model: LARGE System Endianness: LITTLE Mwaitx: DISABLED DMAbuf Support: YES ========== HSA Agents ========== ******* Agent 1 ******* Name: AMD Ryzen 7 5800H with Radeon Graphics Uuid: CPU-XX Marketing Name: AMD Ryzen 7 5800H with Radeon Graphics Vendor Name: CPU Feature: None specified Profile: FULL_PROFILE Float Round Mode: NEAR Max Queue Number: 0(0x0) Queue Min Size: 0(0x0) Queue Max Size: 0(0x0) Queue Type: MULTI Node: 0 Device Type: CPU Cache Info: L1: 32768(0x8000) KB Chip ID: 0(0x0) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 2200 BDFID: 0 Internal Node ID: 0 Compute Unit: 16 SIMDs per CU: 0 Shader Engines: 0 Shader Arrs. per Eng.: 0 WatchPts on Addr. Ranges:1 Features: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: FINE GRAINED Size: 61576816(0x3ab9670) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 2 Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 61576816(0x3ab9670) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 3 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 61576816(0x3ab9670) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: TRUE ISA Info: ******* Agent 2 ******* Name: gfx1032 Uuid: GPU-XX Marketing Name: AMD Radeon RX 6600M Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 1 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 2048(0x800) KB L3: 32768(0x8000) KB Chip ID: 29695(0x73ff) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 2720 BDFID: 768 Internal Node ID: 1 Compute Unit: 28 SIMDs per CU: 2 Shader Engines: 2 Shader Arrs. per Eng.: 2 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 32(0x20) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 32(0x20) Max Work-item Per CU: 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 116 SDMA engine uCode:: 76 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 8372224(0x7fc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 8372224(0x7fc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx1032 Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 ******* Agent 3 ******* Name: gfx90c Uuid: GPU-XX Marketing Name: AMD Radeon Graphics Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 2 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 1024(0x400) KB Chip ID: 5688(0x1638) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 2000 BDFID: 2048 Internal Node ID: 2 Compute Unit: 8 SIMDs per CU: 4 Shader Engines: 1 Shader Arrs. per Eng.: 1 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 64(0x40) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 40(0x28) Max Work-item Per CU: 2560(0xa00) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 471 SDMA engine uCode:: 40 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 4194304(0x400000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 4194304(0x400000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx90c:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 *** Done *** ```

Additional Information

This behavior has been reproduced on MI250, RX6600M, Vega 10, and Ryzen 7 5800H. (gfx90a, gfx1032, gfx900, gfx90c, respectively)

Kernel version: 6.6.22-281.current, with https://github.com/torvalds/linux/commit/96c211f1f9ef82183493f4ceed4e347b52849149 reverted (ref: https://lists.freedesktop.org/archives/amd-gfx/2023-October/100298.html and https://github.com/ROCm/ROCm/issues/2596#issuecomment-1874371163)

LAKostis commented 3 months ago

I can confirm that reverting that commit (https://github.com/ROCm/llvm-project/commit/30a3adf50e2d49dfc97c1b614d9b93638eba672d) fixes the crash with other scenes like classroom but not with blender-3.2 (https://cloud.blender.org/p/gallery/629f23f908e12d4ff15241d3) which still crashed with the similar error (happens only with rocm-6.x):

Compiling HIP kernel ...
hipcc -Wno-parentheses-equality -Wno-unused-value --hipcc-func-supp -O3 -ffast-math --amdgpu-target=gfx1031 -I /usr/share/blender/4.1/scripts/addons/cycles/source --genco /usr/share/blender/4.1/scripts/addons/cycles/source/kernel/device/hip/kernel.cpp -o "/home/lakostis/.cache/cycles/kernels/cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458"
Warning: The --hipcc-func-supp option has been deprecated and will be removed in the future.
Warning: The --amdgpu-target option has been deprecated and will be removed in the future.  Use --offload-arch instead.
Kernel compilation finished in 50.97s.
Read blend: "/home/lakostis/Downloads/Blender 3.blend"
Memory access fault by GPU node-1 (Agent handle: 0x7fd727132600) on address 0x7fd7932ca000. Reason: Page not present or supervisor privilege.
Aborted
GZGavinZhao commented 3 months ago

@LAKostis this is weird, because the blender-3.2 splashscreen renders fine for me on all the devices I've tested. Are you sure your Blender is not using your cycles cache? Maybe try clearing ~/.cache/cycles before every run and install the fatbin files into /usr/share/blender/4.1/scripts/addons/cycles/lib for consistency?

I also see that you seem to be using the Blender provided by your distro. For consistency, can you download the official binaries from Blender and try testing with that instead?

LAKostis commented 3 months ago

@LAKostis this is weird, because the blender-3.2 splashscreen renders fine for me on all the devices I've tested. Are you sure your Blender is not using your cycles cache? Maybe try clearing ~/.cache/cycles before every run and install the fatbin files into /usr/share/blender/4.1/scripts/addons/cycles/lib for consistency?

I also see that you seem to be using the Blender provided by your distro. For consistency, can you download the official binaries from Blender and try testing with that instead?

Yes, I've specially tested this before rocm-6.0.x migration and this demo started failing only with rocm-6.0.x. Cache clearing doesn't help. It can be device specific issue - this demo crashes on my RX 6700 XT (gfx1031) but works on gfx900 (with rendering artifacts and only after setting HSA_ENABLE_SMDA=0)

Regarding the blender build - I'm the blender package maintainer in this distro, so I'm sure what build options where used there :) For the sake of clarity official blender builds crash exactly the same way. I can provide any additional information or logs if you need any.

GZGavinZhao commented 3 months ago

Thanks for the quick response!

It can be device specific issue - this demo crashes on my RX 6700 XT (gfx1031) but works on gfx900 (with rendering artifacts and only after setting HSA_ENABLE_SMDA=0)

Just curious, does compiling against gfx1030 and running Blender through HSA_OVERRIDE_GFX_VERSION on your RX 6700XT change anything? Also what kernel version are you on?

LAKostis commented 3 months ago

HSA_OVERRIDE_GFX_VERSION

No, with HSA_OVERRIDE_GFX_VERSION=gfx1030 it doesn't start with error HIP hipInit: Invalid device

But interesting, it works with lowering -O level:

Read prefs: "/home/lakostis/.config/blender/4.1/config/userpref.blend"
Read blend: "/home/lakostis/Downloads/Blender 3.blend"
Compiling HIP kernel ...
hipcc -Wno-parentheses-equality -Wno-unused-value --hipcc-func-supp -O3 -ffast-math --amdgpu-target=gfx1031 -I /usr/share/blender/4.1/scripts/addons/cycles/source --genco /usr/share/blender/4.1/scripts/addons/cycles/source/kernel/device/hip/kernel.cpp -o "/home/lakostis/.cache/cycles/kernels/cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458"
Warning: The --hipcc-func-supp option has been deprecated and will be removed in the future.
Warning: The --amdgpu-target option has been deprecated and will be removed in the future.  Use --offload-arch instead.
Kernel compilation finished in 50.93s.
Memory access fault by GPU node-1 (Agent handle: 0x7fa997041200) on address 0x7fa9f1b5f000. Reason: Page not present or supervisor privilege.
Aborted
...
❯ blender
Read prefs: "/home/lakostis/.config/blender/4.1/config/userpref.blend"
Read blend: "/home/lakostis/Downloads/Blender 3.blend"
❯ hipcc -Wno-parentheses-equality -Wno-unused-value --hipcc-func-supp -O1 -ffast-math --amdgpu-target=gfx1031 -I /usr/share/blender/4.1/scripts/addons/cycles/source --genco /usr/share/blender/4.1/scripts/addons/cycles/source/kernel/device/hip/kernel.cpp -o "/home/lakostis/.cache/cycles/kernels/cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458"
Warning: The --hipcc-func-supp option has been deprecated and will be removed in the future.
Warning: The --amdgpu-target option has been deprecated and will be removed in the future.  Use --offload-arch instead.
...

with -O1 rendering works. And it start crashing with >=-O2. So something is not right with optimization.

Also what kernel version are you on?

I'm using 6.6.25 kernel + patches up to v6.5-2638-gbf901afac5d5f from amd-staging-drm-next

LAKostis commented 3 months ago

UPDATE: more funny things with compiler:

hipcc -Wno-parentheses-equality -Wno-unused-value --hipcc-func-supp -O1 -ffast-math --amdgpu-target=gfx1031 -I /usr/share/blender/4.1/scripts/addons/cycles/source --genco /usr/share/blender/4.1/scripts/addons/cycles/source/kernel/device/hip/kernel.cpp -o "/home/lakostis/.cache/cycles/kernels/cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458"

This command produces workable kernel despite of warnings about deprecated commands. But this command:

hipcc -Wno-parentheses-equality -Wno-unused-value -O1 -ffast-math --offload-arch=gfx1031 -I /usr/share/blender/4.1/scripts/addons/cycles/source --genco /usr/share/blender/4.1/scripts/addons/cycles/source/kernel/device/hip/kernel.cpp -o "/home/lakostis/.cache/cycles/kernels/cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458"

Produces kernel which crashes. And those kernels are not equal:

-rw-r--r-- 1 lakostis lakostis 4286552 Apr  7 13:10 cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458.crash_O1
-rw-r--r-- 1 lakostis lakostis 3213824 Apr  7 13:09 cycles_kernel_gfx1031_F68CBA054A76B5B26A931C269B2CF458.works_O1