D3D11 varying packing algorithm doesn't take interpolation modifiers into account

What steps will reproduce the problem?
1. Query GL for the max number of varyings, say this equals N.
2. Create a fragment shader and a vertex shader with N + 2 varyings:
   a) N/2 vec3 varyings, smooth interpolation
   b) 1 float varying, flat interpolation
   c) N/2 vec3 varyings, flat interpolation 
   d) 1 float varying, smooth interpolation
3. Try to compile these two shaders. Unless you're lucky, the compilation will 
fail.

HLSL combines adjacent VS outputs into the same register if possible. It can't 
do this if the adjacent outputs use different interpolation modifiers, though. 
Therefore, ANGLE's D3D11 packing algorithm should be smart, and arrange the 
variables such that variables with the same interpolation mode are put next to 
each other in the vertex shader output/pixel shader input. 

As an example, say that N equals 2 for simplicity. Then, in GLSL, define some 
varyings like this:

smooth varying vec3 v1;
flat varying float v2;
flat varying vec3 v3;
smooth varying float v4;

ANGLE should create HLSL shaders that force the interpolation modifiers to 
share the same registers, i.e the VS should output something like this:

float3 v1;
float v4;
nointerpolation float3 v3;
nointerpolation float v2;

Here, v1 and v4 can share one 4-component register, as can v2 and v3.

If it outputs this:

float3 v1;
nointerpolation v2;
nointerpolation v3;
float v4;

then it will end up using 3 registers: one with v1 in it, one with v2 and v3 in 
it, and one with v4. As a result, compilation will fail: there are only 2 
registers available, since N=2.

Original issue reported on code.google.com by aukin...@microsoft.com on 29 Aug 2014 at 7:31

ericmckean / angleproject

D3D11 varying packing algorithm doesn't take interpolation modifiers into account #735