Working around `spirv-cross` by pre-compiling shaders / pre-preparing pipelines

[mitchmindtree]

I recently asked in the gfx riot channel about the feasibility of pre-compiling shaders for each target as an alternative to depending on spirv-cross:

Hi folks! Is it possible to opt-out of using the spirv-cross dependency in favour of pre-compiling the shaders for each target?

@kvark provided some insight:

Basically, there are many factors that affect how the native shader is generated: pipeline layout, specialization, capabilities of the target platform, and other things. The best we can do if you want to run without spirv_cross is: pre-polulate pipeline caches and ship them, add a mode that just returns an error if you are creating a pipeline that is not in the given cache. That path would require a few changes, mostly getting the pipeline cache to a good state (currently, DX12/11 don't implement it).

I've opened this issue so that it's easier to track/discuss this. While I'm at it, I thought I'd add some of our motivation for working around spirv-cross as we transition to gfx for nannou's graphics:

We've had a few people testing our conrod-rendy backend on macOS run into some compilation issues that required updating their xcode toolchain to build spirv-cross properly. We're aiming to remove the need for any of the 3rd party C deps as they (particularly shaderc) have been the source of the vast majority of our issues at the nannou repo over the past couple years and working around them are often the biggest blockers in our creative coding workshops. So far we're managing to avoid shaderc and spirv-reflect by pre-compiling the spirv and specifying layout manually, I think spirv-cross is that last we want to work around before we should have a theoretically smooth story for building on all platforms.

The javelin project looks like a very exciting long-term solution to this, but I wonder if it's worth spending time enabling some feature-gated workaround in the meantime? I'm not sure I'd want to ask to sacrifice time that could have been spent on javelin though :)

[Skepfyr]

It's going to be a while before javelin is ready (but it's great to hear its needed), and I think that precompiled shaders could be useful even with it, as it can't be faster than that.

Interestingly even with spir-v cross enabled it would be nice to allow a cache that falls back to the slow way if it misses.

[freesig]

All spirv-cross is doing is allowing vulkan shaders to be cross compiled to metal. However we could just write the metal shaders by hand for the parts we need. Then compile them on the users mac. Then if they want to do anything special just add the spirv-cross dependency

[kvark]

@freesig writing the metal shader by hand requires you to know:

• all the details about the pipeline layout (and only one layout)
• some specialization constants
• primitive topology (point/non-point)
• supported MSL version

And not only know these but also know how they need to affect your MSL shource, which is an internal implementation detail of the backend (so, no stability guarantees here). I'd say it's not worth trying to go this way, and instead do everything via the standard Vulkan/gfx mechanism: pipeline caches.

[swiftcoder]

See also https://github.com/gfx-rs/gfx/issues/3030 and https://github.com/gfx-rs/wgpu-rs/issues/103 on the same topic.