google / graphicsfuzz

A testing framework for automatically finding and simplifying bugs in graphics shader compilers.
Apache License 2.0
562 stars 117 forks source link

How to confirm the golden image for reference shader? #1105

Closed dubeyabhijeet closed 3 years ago

dubeyabhijeet commented 3 years ago

When we test same shader on ARM and Adreno, sometimes reference image of shader is different. Is there any source of shader which gives us golden image too?

afd commented 3 years ago

No, there isn't. The shaders prefixed stable_ have the property that the images you get from them should be visually indistinguishable - there might be pixel-level differences due to floating-point, but no actual visual glitches.

How are you running GraphicsFuzz - via gfauto, or via the older web ui?

dubeyabhijeet commented 3 years ago

Okay. We are using via older webUI. But we are trying to use GraphicsFuzz as Shader IOT test on different Mobile GPU [Mainly ARM vs Adreno]. We run Shaders with 5 variant on both devices and check if Shaders are rendered correctly or not. But developers have raised genuine concern that how to make sure that Shader is stable. They want golden images of shader to confirm whether ARM GPU has bug or Adreno. Is there any guide for making sure that Shader is stable? Actually we want to run thousands of shaders.

afd commented 3 years ago

Do you mean you want to know which of the sample shaders that ship with GraphicsFuzz are stable, or you want advice about how to know that your own shaders are stable?

Are you testing GLES or Vulkan?

dubeyabhijeet commented 3 years ago

I want advice about how to know that my own shaders are stable. Or any source where we can find stable shaders .

Right now we are using it for GLES.

dubeyabhijeet commented 3 years ago

Another query which is not related to GraphicsFuzz. Iit would be of great help, if you can guide auto tools which can help test Mobile GPU.

afd commented 3 years ago

Regarding your own shaders: you can ensure stability by being very careful with your use of floating-point data. Think about whether a small change in the values your shader uses could result in a large change to the shader's output. For instance, some trigonometric functions are very sensitive to input changes for parts of their input range, and should thus be avoided on such ranges.

Using highp for float and int is generally recommended.

There's no way for getting a golden image because that concept doesn't exist in GLES (the spec is not strict enough). However we find that SwiftShader is a pretty reliable renderer and it can be useful as a second opinion. Intel Mesa is also very reliable.

I don't know of other tools that are useful for helping to test mobile GPUs.

dubeyabhijeet commented 3 years ago

Thanks @afd for the guidance 👍

afd commented 3 years ago

You're welcome.