search

GPUOpen-Drivers / AMDVLK

AMD Open Source Driver For Vulkan
MIT License
1.72k stars 161 forks source link

How to dump LLVM IR before codegen? #249

Closed masahi closed 3 years ago

masahi commented 3 years ago

Hi, with amdPalSettings.cfg and EnablePipelineDump config, I know how to dump shader assembly. But I couldn't figure out how to dump the LLVM IR before the assembly is generated. How do I do it?

For the context, I'm trying to generate packed math instructions like v_pk_fma_f16 via TVM for a DL inference use case. Using the LLVM AMDGPU backend directly (via rocm), I was able to generate nice asm like this https://gist.github.com/masahi/2de1a7dc87e2068ffb50ba6135273f95#file-conv2d_nhwc_float16x2-s-L495-L496. But I couldn't get the equivalent asm if I go through SPIRV and AMDVLK. I only have v_fma_f16 etc, even though the generated SPIRV looks good to me: I have instructions that operate on float16x2 like below. So I want to look at the LLVM IR that AMDVLK generates, to see why I don't get v_pk_fma_f16.

...
        %223 = OpFMul %v2half %217 %222
        %224 = OpFAdd %v2half %214 %223
...
Flakebi commented 3 years ago

Hi, I think the easiest way to debug this is to use the standalone amdllpc compiler. If you built amdvlk yourself, ninja amdllpc will compile build/compiler/llpc/amdllpc. amdllpc can compile pipeline dumps: compiler/llpc/amdllpc -gfxip=10.1 PipelineCs_0x....pipe -v (-v will print the code in some stages) It also supports any LLVM/llc options like -print-after-all, -debug, -stop-before, etc.

masahi commented 3 years ago

@Flakebi Thank you. I think I want to enable EmitLlvm option at https://github.com/GPUOpen-Drivers/llpc/blob/5e02b7edd1b77054bb36b7a33dccd60a63669204/lgc/state/LgcContext.cpp#L292-L296

Is using amdllpc with the option emit-llvm the right way to do it? Or is there a way to enable emit-llvm with amdPalSettings.cfg? (without building the driver or amdllpc from source).

Flakebi commented 3 years ago

It may work if you add this to amdPalSettings.cfg:

LlpcOptions,-enable-outs=1 -shader-cache-mode=0 -include-llvm-ir=1
EnableLog,2

I think this creates a spvLogInfoLlpc file in the dump directory, which should contain LLVM IR.

For amdllpc, there are also pre-compiled binaries in the releases: https://github.com/GPUOpen-Drivers/AMDVLK/releases

The -emit-llvm option will stop compilation before reaching the llvm passes. It is useful to continue compilation with llc or opt. Adding it for the driver will probably make it crash, because no ISA is emitted.

masahi commented 3 years ago

Adding 

LlpcOptions,-enable-outs=1 -shader-cache-mode=0 -include-llvm-ir=1
EnableLog,2

did create spvLogInfoLlpc file but there is no LLVM IR: its content is just

===============================================================================
// LLPC calculated hash results (compute pipeline)

PIPE : 0x518D90E96F2684B9
CS   : 0xA8982029420D564A

===============================================================================
// LLPC calculated hash results (compute pipeline)

PIPE : 0x836B2DF11A491DA8
CS   : 0x2B6A6CB39C329E5B

I'll try amdllpc tomorrow.

masahi commented 3 years ago

I can confirm that amdllpc with -enable-outs option dumps LLVM IR, thanks @Flakebi