Offline shader prototype

[jherico]

Description

Initial work to prototype a solution for #802

The idea here is to take the shader source files that are used by given sample and compile them to SPIRV, and then make them available to the application at build-time rather than requiring any runtime compilation step.

DONE:

• Individual samples can be tagged with "OfflineShaders" and this will cause any SHADER_FILES_GLSL declared in the sample to be targeted for compilation, and will generate the following artifacts
  • A SHADER.debug.spv file that is the result of glslangValidator compilation
  • A SHADER.spv which is the result of running spirv-opt on the above file
  • A SHADER.inl header file which contains the compile SPIRV as a std::vector<uint32_t> (should be changed to std::array<uint32_t, N> so that the structure can be constexpr). The header structure also contains the stage bit and the original source code (which was required because I unwisely picked debug_utils as my test sample, without realizing that it depended on having the source code to attach to the shader object as a tag.

TODO:

• Support shader variants
• Support HLSL
• Figure out a way to declare pipelines and variants in the sample addition code, the way shader files are now for the individual GLSL
• Generate a reflection structure as part of the process that can be used to easily configure descriptor sets
• Make CompiledShader structure constexpr friendly.

General Checklist:

Please ensure the following points are checked:

• [X] My code follows the coding style
• [X] I have reviewed file licenses
• [X] I have commented any added functions (in line with Doxygen)
• [X] I have commented any code that could be hard to understand
• [X] My changes do not add any new compiler warnings
• [X] My changes do not add any new validation layer errors or warnings
• [ ] I have used existing framework/helper functions where possible
• [ ] My changes do not add any regressions
• [ ] I have tested every sample to ensure everything runs correctly

• [ ] This PR describes the scope and expected impact of the changes I am making

  Note: The Samples CI runs a number of checks including:

  • [ ] I have updated the header Copyright to reflect the current year (CI build will fail if Copyright is out of date)
  • [ ] My changes build on Windows, Linux, macOS and Android. Otherwise I have documented any exceptions

Sample Checklist

If your PR contains a new or modified sample, these further checks must be carried out in addition to the General Checklist:

• [ ] I have tested the sample on at least one compliant Vulkan implementation
• [ ] If the sample is vendor-specific, I have tagged it appropriately
• [ ] I have stated on what implementation the sample has been tested so that others can test on different implementations and platforms
• [ ] Any dependent assets have been merged and published in downstream modules
• [ ] For new samples, I have added a paragraph with a summary to the appropriate chapter in the samples readme
• [ ] For new samples, I have added a tutorial README.md file to guide users through what they need to know to implement code using this feature. For example, see conditional_rendering
• [ ] For new samples, I have added a link to the Antora navigation so that the sample will be listed at the Vulkan documentation site

[SaschaWillems]

Just a quick note that we are in the process of completely reworking shader handling in our samples. Please don't invest too much time in this PR

[jherico]

Well that's just awesome. Maybe someone should say something to that effect on the bunch of related open issues. Any of the other open issues I should avoid trying to contribute on, or am I supposed to guess?

[SaschaWillems]

We regularly meet inside Khronos to discuss issues and PRs. We're glad for contributions, but it's always a good idea to ask first before putting too much work into it. As for the shader stuff, see #655, #733 and #846. We also have projects with e.g. a roadmap for samples and the framework, which to our shame we should update so people can see what we're actively working on.

[tomadamatkinson]

Ah I see, there is a reference to the offline shader compiler branch on the issue referenced by this PR. However, I failed to update this to the most recent branch (on my phone now but will be able to drop a link later. It's in the PRs though)

It is essentially to the same point as this PR. It has variant processing but the mechanism could likely be cleaned up.

One of the goals was to have the compile process in python scripts separate to CMake... Variants does make this a little messy.

Feel free to take a look and make any suggestions. There are a few caveats that are discussed in a working group which we are trying to address at the same time

[tomadamatkinson]

Ah great, thanks @SaschaWillems for finding the links!

The WIP branch has grown in complexity as the difficulty isn't particularly in the compiling of shaders. But making sure they are available to the framework.

Granted the framework could be in a better condition to make this easier. Thats something the working group is attempting to address as well

[jherico]

@SaschaWillems sorry if I was snarky. I saw the existing WIP branch was closed and didn't realize it had been replaced with another WIP. I'm trying to participate more in the official samples, and thought I saw an opportunity and this kind of took the wind out of my sails.

@tomadamatkinson

The WIP branch has grown in complexity as the difficulty isn't particularly in the compiling of shaders. But making sure they are available to the framework.

I'm not certain I understand your meaning. You mean making the result of the offline compilation visible to the framework, so that pieces of framework code could reference the compiled versions as easily as the samples?

My approach creates the output artifacts in the build directory in the same relative path as the shader itself, so shaders/foo/bar.vert becomes:

• <BUILD_DIR>/shaders/foo/bar.vert.debug.spv
• <BUILD_DIR>/shaders/foo/bar.vert.spv
• <BUILD_DIR>/shaders/foo/bar.vert.inl

There's no reason that multiple samples or parts of the framework itself can't target the same shaders and share the generated files.

At any rate, is there anything I can contribute to the WIP or anywhere else help is desired (ideally something that uses the C++ bindings, but, you know, whatever).

[SaschaWillems]

@SaschaWillems sorry if I was snarky. I saw the existing WIP branch was closed and didn't realize it had been replaced with another WIP. I'm trying to participate more in the official samples, and thought I saw an opportunity and this kind of took the wind out of my sails.

I didn't percieve it as snarky ;) We should do a better job at communicating what we're working on. We have two projects in this repo (https://github.com/KhronosGroup/Vulkan-Samples/projects?type=classic), but we haven't really updated those, and we should do that to avoid people wasting their time.

Most external contributions usually come in the form of samples. As Tom noted, we're heavily reworking the framework and that includes how we handle shaders. We also have companies that do on these samples, making it kinda hard to coordinate with external contributions.

[tomadamatkinson]

I'm not certain I understand your meaning. You mean making the result of the offline compilation visible to the framework, so that pieces of framework code could reference the compiled versions as easily as the samples?

Vulkan Samples effectively has 3 frameworks. Api Samples, Performance Samples (C) and Performance Samples (C++). The Api Samples framework is lightweight and easy to work with - but is glued on top of Performance (C). The Performance Samples framework is a little more robust but also opinionated on how it is used. Components like the Resource Cache and Shader Modules come into play.

The last issue I ran into with the WIP branch was trying to integrate the same loader into all three frameworks effectively. (Granted I got a little sidetracked staring at the resource cache for too long 😄). It's been a few weeks now since I last got a chance to take a look at it and so would appreciate some help getting it over the line if you are interested.

[jherico]

Thanks. I'll check out the branch, look at the PR and try to wrap my head around the changes and what's remaining. It's been a while since I worked directly with the framework code, so I want to make sure I understand both the current state and where you're trying to take it.