[feature] Shader code testing instruction and examples

[TotalKrill]

Shader code testing procedures that includes mobile devices

Some instructions or help on how to test shader code generated by naga-oil, or the wgsl code in the codebase in general. A lot of the functions in the shader files could really benefit from some way of testing them on real hardware. Since reading this from Unity states the need to test shader code that is supposed to run on mobile devices, where the hardware capabilities and performance features can lead to numerical issues.

Many of the currently open issues regarding crashes, issues on mobile devices, can probably be traced to shadercode that behaves differently on mobile units.

The PC GPUs, according to the unity docs, behave much more coherently, but the code itself would very much benefit from some unit testing in anyway.

[Elabajaba]

There's a daily CI job that runs bevy on real mobile devices (ios and android), and compares a screenshot to see if it works. (eg. https://github.com/bevyengine/bevy/actions/runs/6468999093)

edit: I'm not sure what the usage limits are, so I don't know if we could test a heavier example or anything.

[TotalKrill]

That is always something, but it would be even nicer if the shader code could be unit tested someway.

Like cargo test shader_test_pbr_opaqueness --target shader_target

I understand that is not really feasible, but then something like

• cargo build shadercodes // would output the shadercodes as glsl, or the different codes
• send compiled shadercode to gpu on device
• get response somehow, probably a picture or smth

• check some known pixels value, and that it is inside known threshold, or just get pure numerical values back that we could check

  ideally a script that does this, probably something that the current pipeline could be highjacked to do, but instead of testing the entire renderer, have the goal of unit testing the smallest unit of compiled shaders and then output the results.

  Unfortunately, I am personally way to lost in this entire process to even know where to begin. But I suspect that having this available, could stop regressions in the renderer on different device much quicker.

[mockersf]

This is not very hard to do for each shader / case, but it gets hard to do generically for every path in your shader. The shader code use whatever inputs you give it, it doesn't work as an abstract function. So you actually have to prepare a scene with things for it to render.

And the biggest blocker for this is that you'll need to have the actual device with the gpu you want to test available.

For Bevy case, the mobile example is a very limited case of going through a few shaders (well, pbr and UI). With the help of https://www.browserstack.com, we're able to run that example on a few real devices in a GitHub action. The mobile example could be improved to test more parts of the pbr shader (skinning for example), or the sprite rendering shader. We already know the shadow part crashes on some android devices...

For your own game, as an indie dev it's very unlikely you'll have access to a mobile test farm, so either you only test on the few devices you have on hand, or you'll have to pay for the service...