search

microsoft / DirectXShaderCompiler

This repo hosts the source for the DirectX Shader Compiler which is based on LLVM/Clang.
Other
3.04k stars 677 forks source link

[SPIRV][Question]Link libraries #2633

Open Vincent-P opened 4 years ago

Vincent-P commented 4 years ago

Hi, I saw that SPIRV have a Linkage capability and can import and export functions or global variables. SPIRV-Tools also has an experimental linker to combine spirv files.

So is it possible to compile a module with the lib_6_3 target profile that exports some functions, and import those in a separate file?

jaebaek commented 4 years ago

@Vincent-P Thank you for opening this issue. As far as I know, we do not generate relocatable code, but I am not 100% sure. I will get back to you after researching on it.

jaebaek commented 4 years ago

Hi @Vincent-P , unfortunately, we currently do not support a compilation of a relocatable code. lib_6_3 and lib_6_4 are used only for the ray tracing code but used for general code. For the relocatable code generation, we may need some big changes in the source code.

In addition, as far as I know, spirv-link is also work-in-progress. See here.

Vincent-P commented 4 years ago

Okay, thanks for your answer!

gongminmin commented 4 years ago

Any plan to support lib_6_x profile?

ehsannas commented 4 years ago

@gongminmin I'm wondering what specifically you are referring to? Is there a document/reference that you can point us to?

If you're referring to using [shader(...)] for raytracing purposes, it is supported (example).

If you're referring to a more broad scope, can you please provide an example and/or the lib6* doc/spec?

gongminmin commented 4 years ago

Currently HLSLs can be compiled to Dxil modules as general purpose lib_6_x, and linked to pixel shader, vertex shader, etc. (https://github.com/microsoft/DirectXShaderCompiler/blob/master/tools/clang/unittests/HLSL/LinkerTest.cpp). Can SPIR-V back-end do the same thing? Thanks.

ehsannas commented 4 years ago

IIUC the linking stage is also done offline (via DXC), not at runtime. right?

ehsannas commented 4 years ago

Thanks for the reference. The SPIR-V spec has the Linkage capability. At a high level, to get that to work we'd need do a few things: 1) the SPIR-V backend in DXC to generate proper linkage information. 2) spirv-link inside SPIRV-Tools currently only works for OpenCL, so there's some work that needs to be done there in order to properly link binaries and generate a SPIR-V that's valid for Vulkan. 3) Since SPIRV-Tools is already included as a dependency, we should be able to use it to link multiple spir-v binaries and generate one valid SPIR-V binary.

ehsannas commented 4 years ago

This is something that we should probably invest time in since we want to create a similar experience for the HLSL developers targeting Dxil and SPIR-V.

[side: I haven't dived into the Vulkan spec to check the finer details. There might be something in there that makes this effort more difficult/impossible ...]

gongminmin commented 4 years ago

Yes, the link is done by dxc. One advantage of compile + link is that it's way faster than concatenate source code strings together and compile it. One application of shader linking in old D3DCompiler is compile shader fragments in develop time, and link fragments in runtime. However, D3DCompiler's linking doesn't have link time code generation. The linked shader has lower efficient compare to concatenate strings way. The other disadvantage of shader linking in D3DCompiler is it can't declare a function in one module, and implement it in another module. DXC's linker doesn't have these problems, it should be much better to be used as compile + link compare to D3DCompiler's.

But I just found that currently DxcLinker is a Windows only thing (Line 115 of tools/clang/tools/dxcompiler/dxcapi.cpp). More works need to be done for extending it to a cross-platform component.

devshgraphicsprogramming commented 1 year ago

Maybe there's could be a "poor man's linking" option for SPIR-V which would be to merge/concatenate the ASTs from multiple files, essentially an equivalent of a C++ Unity Build.

devshgraphicsprogramming commented 11 months ago

Maybe there's could be a "poor man's linking" option for SPIR-V which would be to merge/concatenate the ASTs from multiple files, essentially an equivalent of a C++ Unity Build.

well I guess that would just be #5771

s-perron commented 1 month ago

It would be possible for us to implement some type of linking.

We could add the "import" decoration to undefined function. In https://godbolt.org/z/4s8xaEdTK, foo would be an import function instead of issuing an error. Then spirv-link (part of spir-v tools) could be used to link with spir-v module where foo is an export funcution. Note that linking would have to occur before passing the spir-v to the driver because Vulkan does not accept the linkage attribute.

This would seem to be inline with section 8.8 in https://microsoft.github.io/hlsl-specs/specs/hlsl.pdf.

However, we might have problems linking global variables. At the footnote says,

These are not really linked with other translation units but rather their values are loaded indirectly based on cbuffer mapping.

This does not map cleanly to anything in spir-v.

The other problem we might have is with backwards compatibility. Developers might use lib_6_x to build a shader module with multiple shaders in them because that is the only way they can. If we start adding the linkage decorations for everything that should be export, they will start to get module that have the linkage attribute that will have to be stripped before the can pass it to the Vulkan driver.