Complete HLSL -> SPIR-V translator

[johnkslang]

Requested HLSL features. If it's checked, it's been implemented and working for some workload. If a checked feature is not working correctly, there should be an issue reporting the incorrect behavior. A missing feature should be requested here.

• [x] geometry, domain, and hull shaders
• [x] member methods inside a structure, see issue #535
• [x] full pass through of attributes on loops/if/switch, e.g., [unroll], [flatten], etc., see issue #836
• [x] Maphalf to float. See issue #492.
• [x] namespace (about half done)
• conversions
  • [x] structure cast of scalar: PS_OUTPUT __output__ = (PS_OUTPUT)(scalar)
  • [ ] inout arguments needing bidirectional conversion
  • [x] smearing implicit-conversion
  • [x] vector truncation conversion
  • [x] vector of size 1 vs. scalar
• [x] Basic RWBuffer support
• [x] deference of textures
• [x] arbitrary (structure) texture return-type, see issue #569.
• [x] sub-vec4 return types from sampling
• compute shaders
  • [x] numthreads
  • [x] SV_DispatchThreadID -> gl_GlobalInvocationID
  • [x] groupshared -> shared
• HLSL-specific preprocessing
  • [ ] trailing () parens are required/prohibited with an empty argument list in #defines
  • [x] #include support
  • [x] -D and -U for macros on command line
• [x] class keyword, member functions, variables, static and non static
• [ ] class inheritance from interface see https://msdn.microsoft.com/en-us/library/windows/desktop/ff471421(v=vs.85).aspx
• [x] Multi-dimensional arrays, ala float myfloat[4][5];
• [x] minN types, ala https://msdn.microsoft.com/en-us/library/windows/desktop/bb509646(v=vs.85).aspx.
• [x] vertical path hookup through to SPIR-V generation
• [x] generic preprocessor (same as GLSL)
• [x] scanner (keywords, identifiers, etc.)
• [x] basic types: all int/uint/float/double scalar/vector/matrix
• [x] basic recursive-decent language grammar: declarations, expressions, statements, loops, if-else, functions
• [x] parsing of semantics, annotations, registers, etc. on declarations
• [x] parsing of attributes on loops/if/switch, e.g., [unroll], [flatten], etc.
• [x] basic symbol-table scoping
• [x] switch statements, { } initializers, typedef
• [x] templatized vector/matrix/etc. declaration grammar
• [x] base set of all intrinsics mappable to existing operators
• [x] HLSL-specific intrinsics, mapped to new operators
• [x] vec1/mat1xN/matNx1-based, etc.
• [x] DX10 style sampler declarations
• [ ] immediate (literal) samplers
• [ ] DX9 style sampler declarations
• [x] 2DMS and Buffer texture support
• [x] method-based image/texture lookup
• [x] cbuffer/tbuffer
• [x] register and packoffset for buffers
• [x] in/out/inout qualifiers, e.g, void MyFunc(out float x)
• [x] default parameters, e.g, void MyFunc(float x=3)
• [x] overloaded function signature selection under implicit type conversion
• DX12 and beyond
  • [x] syntax: ConstantBuffer
  • [x] Wave Query,Wave Vote,Wave Broadcast,Wave Reduction,Wave Scan and Prefix,Quad-wide Shuffle operations,etc..
  • [x] 64-bit types.
• [x] printf

[ghost]

[ ] packoffset?

The "misc pending intrinsics" line above includes both the "unusual" intrinsics like printf as well as ones that accept either 1-vectors or Nx1 mats, which can be added, but aren't yet available.

[oscarbg]

also would be nice if you can add DX12 HLSL support.. parsing RootSignature,etc.. for ex. I see in: https://github.com/Microsoft/DirectX-Graphics-Samples miniengine AdaptExposureCS.hlsl shader

[RootSignature(PostEffects_RootSig)]

define PostEffects_RootSig \

"RootFlags(ALLOW_INPUT_ASSEMBLER_INPUT_LAYOUT), " \ "CBV(b0, visibility = SHADER_VISIBILITY_VERTEX)," \ "CBV(b0, visibility = SHADER_VISIBILITY_PIXEL)," \ "DescriptorTable(SRV(t0, numDescriptors = 1), visibility = SHADER_VISIBILITY_PIXEL)," \ "StaticSampler(s0, visibility = SHADER_VISIBILITY_PIXEL," \ "addressU = TEXTURE_ADDRESS_CLAMP," \ "addressV = TEXTURE_ADDRESS_CLAMP," \ "addressW = TEXTURE_ADDRESS_CLAMP," \ "filter = FILTER_MIN_MAG_MIP_LINEAR)"

[ghost]

[ ] in/out/inout qualifiers, e.g, void MyFunc(out float x);. Widely used - I have local code for this which I'll submit a PR for. Edit: #385.

[ ] Default parameters, e.g, void MyFunc(float x=3); but maybe this shouldn't move to the main list without a concrete use case.

[ghost]

DX10 style sampler declarations [in progress]

The only things missing there now are arguably orthogonal features such as sub-vec4 returns and 2DMS/Buffer, which is currently missing everywhere and touches a lot besides the decls. Otherwise, DX10 texture decls are working. (DX9 is still TODO, and the methods are still in progress).

[ghost]

conditional expressions (a ? b : c ternary operator).

(Edit for strikethrough now that this is in the main list)

[ghost]

I think this can be marked as done, modulo a few things with no feature mappings:

method-based image/texture lookup

DX9 texturing is still a TODO.

[ghost]

• Multi-dimensional arrays, ala float myfloat[4][5];
• minN types, ala https://msdn.microsoft.com/en-us/library/windows/desktop/bb509646(v=vs.85).aspx.

RWTextures/Buffers is in progress working.

[Alprog]

One of the most exciting feature of hlsl against glsl is ability to write several entry points in one file. And SPIR-V support this feature too. But right now you need to explicitly set only one entry point. I've implemented function addEntryPoint(std::string name, EShLanguage stage) but I've gotten multiple issues with semantic parsing. It's tricky to workaround all cases. Do you plan to support multiple entrypoints?

[johnkslang]

This has come up in a couple places. First, an HLSL file with multiple entry points is really only valid to parse a single entry point at a time, because of the semantic changes. Second, there should be an independent modular tool for taking two SPIR-V modules and merging them into a single module.

Combined, these two points indicates the correct design is to independently compile a source file, once for each entry point, making a set of SPIR-V modules, and then use a SPIR-V merging tool to merge multiple modules into a single module.

[Alprog]

because of the semantic changes

I don't understand where ambiguity comes from. Can you give me example?

For instance, I use shaders like this:

cbuffer ConstantBuffer : register(b0)
{
    row_major float4x4 MVP;
};

struct Vertex
{
    float3 position : POSITION;
    float4 uv : TEXCOORD;
};

struct Gradient
{
    float4 position : SV_POSITION;
    float2 uv : TEXCOORD;
};

Texture2D g_texture : register(t0);
SamplerState g_sampler : register(s0);

Gradient vsmain(Vertex v)
{
    Gradient result;  

    result.position = mul(float4(v.position, 1), MVP);
    result.uv = v.uv; 

    return result;
}

float4 psmain_normal(Gradient in) : SV_TARGET 
{   
    return g_texture.Sample(g_sampler, in.uv);
}

float4 psmain_grayscale(Gradient in) : SV_TARGET 
{   
    float4 color = g_texture.Sample(g_sampler, in.uv);
    color.rgb = color.r * 0.3 + color.g * 0.59 + color.b * 0.11;
    return color;
}

[johnkslang]

If you compiled this three times, once for each entry point, and got three SPIR-V modules, which could potentially then be merged together, would that work?

The problems comes in if one potential entry point calls another, which is supported and happens:

float4 A(...) : SV_Position { }
float4 B(...) : SV_Target { return A(...); }

The semantics of I/O for A are different based on whether A or B is the entry point, and they can't be entry points at the same time. During compilation, the compiler has to know whether A is an entry point or not. However, the shader can be compiled twice, once with A as the entry point and once with B.

[Alprog]

Yeah, it works. But it increase compilation time x3. Moreover, it require additional tool (is it exist?). Situation mentioned by you is unable to compile with several entrypoints, right. And it's ok to produce invalid code in this situation. But it is rare case. Why not to support multiply entrypoints when it is possible?

[hrydgard]

Small feature request: Support

#pragma pack_matrix( row_major )

to default to row major matrices. Right now the pragma seems to be without effect.

[hrydgard]

Two more questions:

Are geometry shaders supported? I can't seem to get them to parse. #590

Additionally, is there any way to declare textures in HLSL to get combined texture/sampler objects in the SPIR-V? I need those for later spirv-cross SPIR-V->GLSL code generation, it seems...

[Alprog]

Texture2D g_texture : register(t0); SamplerState g_sampler : register(s0);

This works well. Both in spir-v, and in glsl (I use spirv-cross).

[hrydgard]

Yeah, oops - I got it working too right after posting, I just forgot to call build_combined_image_samplers in spirv-cross. Though, that still creates separated sampler/texture in SPIR-V which is not optimal in Vulkan... Also need to look into how the numbering will work for that.

[ghost]

Are geometry shaders supported? I can't seem to get them to parse

Not yet, but seems like a good thing for the list above, along with DS/HS.

There is some CS support (e.g, numthreads).

[ghost]

BTW, this item:

minN types

Can be marked as done (in #570).

[Alprog]

It seems like I made multiple entypoint processing. Strategy:

1. Add several TShader to TProgram with same hlsl but different entrypoint.
2. Change link process: allow multiple entries and resolve global bodies conflict (prefer entrypoint version).
3. Store all entries data in one TIntermediate.

[ghost]

:: scoping operator. It seems low priority, but this is valid:

int myfn() { return 42; }
void foo() { int myfn = ::myfn(); }

[ghost]

Under the compute shader section, this one is done as of #572:

• numthreads

[Joojooo]

Sorry for asking. When a feature from the above list is checked, does it mean that it's been implemented, or only that it's been requested and is among the priorities?

[johnkslang]

If it's listed, it's a requested feature. If it's checked, it's been implemented and working for at least some workloads. If a checked feature is not working correctly, there should be an issue reporting the incorrect behavior.

I'm about to revisit the list and all HLSL issues to ensure they agree with this.

[Joojooo]

Thanks for the precision.

I was asking because after I tried to parse an HLSL shader defining a sampler state, for example: SamplerState LinearSampler { Filter = MIN_MAG_MIP_LINEAR; };

I received the following warning: "WARNING: 0:10: 'immediate sampler state' : unimplemented" However in the list I noticed that "DX10 style sampler declarations" is checked

[johnkslang]

I added immediate samplers to the list.

[ghost]

@Joojooo: at the moment glslang parses them and ignores them (hence the warning in your quote), since there's not much else to be done. AFAIK there's no extant driver mechanism in Vulkan to set these states up from data passed back from a shader text. So, it's unimplemented, but also about as implemented as it's going to get short of such a mechanism :).

[Joojooo]

I see, thanks a lot for the details!

[shmerl]

Pretty major development: https://github.com/Microsoft/DirectXShaderCompiler

Will it help with HLSL translation to GLSL or SPIR-V?

[ghost]

unusual intrinsics: printf, vec1/mat1xN/matNx1-based, etc.

About that one: there are Nx1 and 1xN mats now, and associated intrinsic overloads accepting mats.

Printf is still not available though, nor noise, msad4, errorf, and maybe a few others.

[ghost]

geometry, domain, and hull shaders

Can probably mark as implemented, and defects can be logged if and when they appear.

[ghost]

syntax: ConstantBuffer

Can be marked as done.

[ClemensRoegner]

Hi!

I found that texture intrinsic functions are not working when having a texture object with just one component. Example (based on altering the hlsl.getdimensions.dx10.frag in the test folder):

Texture2DMS <float> g_tTex2dmsf1;
....
g_tTex2dmsf1 . GetDimensions(WidthU, HeightU, NumberOfSamplesU);

fails with something like

hlsl.getdimensions.dx10.frag:150: 'GetDimensions' : no matching overloaded function found 
ERROR: 1 compilation errors.  No code generated.

I did some digging and the issue seems to be that the intrinsic table does not have the function signetures required for scalar textures (aka shadow/depth textures). In other words: the table provides something like GetDimensions(t2M1;u1;u1;u1; but the code wants to have GetDimensions(tS2M1;u1;u1;u1;. Hower, I couldnt figure yet out where to add the necessary permutation. Any hints are welcome :)

Cheers, Clemens

P.S.: Same is true for .Load etc.

[ghost]

Hi @ClemensRoegner,

Could you submit an issue for this with a compilable example (so we can discuss under its own thread)? I tried the code below which compiles cleanly for me in current master (136b1e2d5d90284fd7bdd77ed605c70a8d31c8c4). Once I have the magic sauce to duplicate it, I'll have a look and see what's going wrong. It looks like there is a hole in the current test coverage around texture intrinsics with sub-vec4 templatized texture types, so maybe something has fallen through the cracks...

thanks!

Texture2DMS <float>  g_tTex2dmsf1;
Texture2DMS <float2> g_tTex2dmsf2;
Texture2DMS <float3> g_tTex2dmsf3;
Texture2DMS <float4> g_tTex2dmsf4;

float4 main()
{
    uint MipLevel;
    uint WidthU;
    uint HeightU;
    uint ElementsU;
    uint DepthU;
    uint NumberOfLevelsU;
    uint NumberOfSamplesU;

    g_tTex2dmsf1 . GetDimensions(WidthU, HeightU, NumberOfSamplesU);
    g_tTex2dmsf2 . GetDimensions(WidthU, HeightU, NumberOfSamplesU);
    g_tTex2dmsf3 . GetDimensions(WidthU, HeightU, NumberOfSamplesU);
    g_tTex2dmsf4 . GetDimensions(WidthU, HeightU, NumberOfSamplesU);

    g_tTex2dmsf1 . Load(int2(1,2), 3);
    g_tTex2dmsf2 . Load(int2(1,2), 3);
    g_tTex2dmsf3 . Load(int2(1,2), 3);
    g_tTex2dmsf4 . Load(int2(1,2), 3);

    return 0;
}

[ClemensRoegner]

Yeah, sorry I checked the glslang version again and it seems like the old one (distributed with the vulkan sdk) still has the problem. Not the current one. Terrible sorry for the fuzz.

[ghost]

@ClemensRoegner - ok, glad it's OK now!

[ghost]

Will support of conversion HLSL shader model 6.0 to SPIRV with extensions?

[ghost]

arbitrary (structure) texture return-type, see issue #569.

The line item text above sounds like it's talking about returning structures from texture fetch, ala "Texture2D foo;", but #569 was talking about StructuredBuffer. I've been thinking of those as independent features with little shared implementation. Maybe there should be two line items?

I think StructuredBuffers with user struct types is already implemented, but textures of user structs are not. (They support vectors and scalars of basic types as template parameters so far, but HLSL also allows templatizing on user structs whose members must be of the same basic type, with any mix of vectorness up to 4 total components). I think it can be added in a lightweight way without having to jam a whole type object into a TSampler.

I could take a swing at that. It's probably just down to luck that no workload has wanted it yet. Edit: in progress as #1008 #1017. Probably testable now, but code still being worked on. Done

[ratchetfreak]

Looks like microsoft is also doing a hlsl->spir-V compiler using their open source hlsl compiler: https://github.com/Microsoft/DirectXShaderCompiler/blob/master/docs/SPIR-V.rst

[ceztko]

Can someone clarify if there will be any relationship/collaboration between DirectXShaderCompiler and Khronos to make HLSL first citizen in Vulkan? From the outside it seems there's a race between the two projects, that also have incompatible approaches in some aspects. I've also the feeling, but this is just my opinion, that glslang will always struggle behind with regard to feature parity with HLSL, carrying potentially more bugs than the counterpart and also risks of regressions in glsl because of the clutter necessary to support substantially different languages.

Also another question: with the announce in Vulkan 1.1 of the support for HLSL do Khronos specifically refers to the implementation in glslang or they leave it open to other implementations too?

[johnkslang]

Can someone clarify if there will be any relationship/collaboration between DirectXShaderCompiler and Khronos to make HLSL first citizen in Vulkan?

The only language that is really first class in Vulkan is SPIR-V. SPIR-V is made to support lots of high-level language projects, like glslang GLSL, glslang HLSL, the DXC-based SPIR-V back end, etc.

Khronos-level extension support for HLSL is for HLSL in general, across multiple potential front ends.

From the outside it seems there's a race between the two projects,

Both projects are underway, following their natural evolution, with different strengths/weaknesses, levels of contributions, etc. Neither will see support cut off. Contributions will always be accepted, pending review, etc.

that also have incompatible approaches in some aspects.

These would be nice to converge, where they exist and can be converged, and have resources to converge them. Specific issues will help.

I've also the feeling, but this is just my opinion, that glslang will always struggle behind with regard to feature parity with HLSL,

Yes, that's probably true, for new features that show up first in MS front-ends.

carrying potentially more bugs than the counterpart and also risks of regressions in glsl because of the clutter necessary to support substantially different languages.

Actually, there is very little risk of bugs to GLSL due to HLSL support. There really is not an issue here.

Also another question: with the announce in Vulkan 1.1 of the support for HLSL do Khronos specifically refers to the implementation in glslang or they leave it open to other implementations too?

Khronos does not pick a favored HLSL front end. Other HLSL front ends are welcome to participate, and have different strengths/weaknesses.

[ghost]

Also, from #1346: "Support 64-bit integers" (part of SM6).

[johnkslang]

The SM6 wave stuff was added.

[ghost]

The SM6 wave stuff was added.

Right... I was thinking more about 64-bit support outside the wave intrinsics. E.g, you can pass them to various other random intrinsics like abs(), so the protos should get expanded, and so forth. Seems like mostly a separate thing from Wave*, but if it's subsumed under that bucket, then good enough.

[johnkslang]

Agreed, I only meant the wave part of SM6 was added, not 64-bit in general.

[ClemensRognerSD]

Hi!

Regarding immediate (literal) samplers: 1) are there plans to implement them? 2) are they even possible? native spirv does not have that functionality and looking at the glslang code does not indicate it is there hidden away.

On the topic of dx9 sampler declaration: 3) Leaving the literal sampler declaration aside, essentially a dx9 sampler is equivalent to the spirv sampler as it combines texture and sampler to be set from the api, right?

Cheers, Clemens

[johnkslang]

About 1, no plans, and 2, right, the platform can't support it. There might be a scheme of indexing all literal possibilities and having the driver set them all up, to emulate it. Would maybe depend on what DXC does.

3, sounds likely, but that path is off/not supported.

[vaspoul]

One issue I came across with namespace is that struct declarations aren't propertly scoped. The following results in an 'redefinition' error

struct Foo { float4 param; };

namespace Bar { struct Foo { float4 param; }; }

[untodesu]

DX9 style sampler declarations

AFAIK they are pretty similar to the existing GLSL ones:

sampler2D ms : register(s0); // layout(binding = 0) uniform sampler2D ms;
float4 main(in float2 uv : TEXCOORD0) : COLOR // or SV_TARGET, whatever
{
    return tex2D(ms, uv); // texture(ms, uv);
}

Thus it shouldn't be that hard to implement since the existing code for GLSL is present around there...

[ukari]

how about using main as a default entry point for hlsl, just like what shaderc/glslc does?

doc shaderc/glslc

4.5.6. -fentry-point=<name>

-fentry-point=<name> lets you specify the entry point name. This is only significant for HLSL compilation. The default is "main".