Optional stage fallback when stage could not be detected

[hsimpson]

This problem is occurring again for me with .glsl files.

Originally posted by @Nolram12345 in https://github.com/hsimpson/vscode-glsllint/issues/3#issuecomment-1005089053

[Cazadorro]

This would be very helpful to me. It's common, almost standard, to use the .glsl extension with out any proceeding extension for library files which are to be included in other files. ie, utils.glsl. which are then included:

#include "utils.glsl" Right now I have to put .comp in my extension, but that shouldn't be necessary.

[hsimpson]

Agreed, but I have to investigate a more smart way, because some one could use .glsl also for a fragment, vertex or compute shader. And this is very tricky. What is the problem to use the compound extensions like .comp.glsl, .vert.glsl or .frag.glsl? Does this break some other workflow?

My main problem is, that I need to know the stage (vert, frag, comp, others) 100% for sure before sending the shader code to glslangValidator binary.

See glslangValidator --help for additional Information.

[Cazadorro]

What is the problem to use the compound extensions like .comp.glsl, .vert.glsl or .frag.glsl? Does this break some other workflow?

Yes, because it causes our tools to try to compile them as if they are each of those types of shaders. They aren't, and thus they fail to compile on our side if we do this. This is also not something we can fix, other features rely on this happening.

here's an example of what one of our .glsl files look like:

#ifndef BASICREFTYPES_GLSL
#define BASICREFTYPES_GLSL

layout(buffer_reference, buffer_reference_align=4, scalar) buffer float32_array{
    float32_t data[];
};
layout(buffer_reference, buffer_reference_align=8, scalar) buffer f32vec2_array{
    f32vec2 data[];
};
layout(buffer_reference, buffer_reference_align=16, scalar) buffer f32vec3_array{
    f32vec3 data[];
};
layout(buffer_reference, buffer_reference_align=32, scalar) buffer f32vec4_array{
    f32vec4 data[];
};

layout(buffer_reference, buffer_reference_align=8, scalar) buffer float64_array{
    float64_t data[];
};
layout(buffer_reference, buffer_reference_align=4, scalar) buffer f64vec2_array{
    f64vec2 data[];
};
layout(buffer_reference, buffer_reference_align=4, scalar) buffer f64vec3_array{
    f64vec3 data[];
};
layout(buffer_reference, buffer_reference_align=4, scalar) buffer f64vec4_array{
    f64vec4 data[];
};

layout(buffer_reference, buffer_reference_align=1, scalar) buffer uint8_array{
    uint8_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u8vec2_array{
    u8vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u8vec3_array{
    u8vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u8vec4_array{
    u8vec4 data[];
};

layout(buffer_reference, buffer_reference_align=2, scalar) buffer uint16_array{
    uint16_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u16vec2_array{
    u16vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u16vec3_array{
    u16vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u16vec4_array{
    u16vec4 data[];
};

layout(buffer_reference, buffer_reference_align=4, scalar) buffer uint32_array{
    uint32_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u32vec2_array{
    u32vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u32vec3_array{
    u32vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u32vec4_array{
    u32vec4 data[];
};

layout(buffer_reference, buffer_reference_align=8, scalar) buffer uint64_array{
    uint64_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u64vec2_array{
    u64vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u64vec3_array{
    u64vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer u64vec4_array{
    u64vec4 data[];
};

layout(buffer_reference, buffer_reference_align=1, scalar) buffer int8_array{
    int8_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i8vec2_array{
    i8vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i8vec3_array{
    i8vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i8vec4_array{
    i8vec4 data[];
};

layout(buffer_reference, buffer_reference_align=2, scalar) buffer int16_array{
    int16_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i16vec2_array{
    i16vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i16vec3_array{
    i16vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i16vec4_array{
    i16vec4 data[];
};

layout(buffer_reference, buffer_reference_align=4, scalar) buffer int32_array{
    int32_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i32vec2_array{
    i32vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i32vec3_array{
    i32vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i32vec4_array{
    i32vec4 data[];
};

layout(buffer_reference, buffer_reference_align=8, scalar) buffer int64_array{
    int64_t data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i64vec2_array{
    i64vec2 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i64vec3_array{
    i64vec3 data[];
};
layout(buffer_reference, buffer_reference_align=1, scalar) buffer i64vec4_array{
    i64vec4 data[];
};

#endif //BASICREFTYPES_GLSL

My main problem is, that I need to know the stage (vert, frag, comp, others) 100% for sure before sending the shader code to glslangValidator binary.

Most of our code in .glsl is not specific to vert, frag, or compute, and is readily used between all of them, so even if you always used compute for this, it would be better than nothing, and work for 90% of our work flow, vs 0% right now. Alternatively you could run it through as compute and then as frag, and validate with the one that works, there's very little that's frag specific (quad subgroup access is one common thing), and nothing we've used is vert specific.

Alternatively again, you could only validate .glsl when it is included in another file, run it as said type of shader.

[hsimpson]

Would in your case a configurable fallback to comp stage ok? So every shader file which is not a dedicated stage like vert/frag would be used as comp?

[Cazadorro]

Yes, comp as default should work for most scenarios scenarios for us.

[hsimpson]

So I will implement a configurable fallback stage which is empty per default. But I won't get around to implementing that until next week.

[hsimpson]

I am trying to implement the fallback, but I could not get you shader to lint with glslangValidator I have some problems with the buffer_reference which looks like is only possible for Vulkan. Could you test yourself to get the file checked correctly with glslangValidator and help me out with the parameters.

[Cazadorro]

That file will only work when included in a file with the appropriate extensions enabled (buffer reference 2, all those types, and scalar block layout), so if validated on it's own, it probably won't work. glslangValidator works with vulkan (infact, is coded against the vulkan GLSL first, then has switches internally to handle other languages), so I'm not sure why vulkan support would be an issue.

#extension GL_EXT_scalar_block_layout: enable
#extension GL_EXT_buffer_reference2 : enable
//for uint64_t etc...
#extension GL_EXT_shader_explicit_arithmetic_types         : enable
#extension GL_EXT_shader_explicit_arithmetic_types_int8    : enable
#extension GL_EXT_shader_explicit_arithmetic_types_int16   : enable
#extension GL_EXT_shader_explicit_arithmetic_types_int32   : enable
#extension GL_EXT_shader_explicit_arithmetic_types_int64   : enable
#extension GL_EXT_shader_explicit_arithmetic_types_float16 : enable
#extension GL_EXT_shader_explicit_arithmetic_types_float32 : enable
#extension GL_EXT_shader_explicit_arithmetic_types_float64 : enable

Some of these extensions are only available in vulkan.

[hsimpson]

Thanks, I can successfully link your file with a few changes:

• Using the new fallback option (not yet released)
• Enable the specific extensions
• Adding #version 460 as a the first line
• Adding -V as additional args.

I have created a test release, could try this on your side please? If everything works, then I will merge this to into master and release it to the marketplace. To test this build go to https://github.com/hsimpson/vscode-glsllint/releases/tag/1.8.0-test and download the vscode-glsllint-1.8.0.vsix file, then install it manually into VSCode. After this configure these both options:

"glsllint.glslangValidatorArgs": "-V",
"glsllint.fallBackStage": "comp"

In your VSCode settings. Then restart VSCode and try if your shaders are linting successfully.

[Cazadorro]

It appears to work (getting around the problems below temporarily, I get correct inline errors in VSCODE for included files)? We have a few problems though:

• glslc does not require GL_GOOGLE_include_directive, infact if you try to enable it again, glslc fails to work. There might be an arg for enabling this extension by default but IDK, the workaround would be, for every single file, place a macro that would be conditionally enabled by glslangvalidator. I don't think this is a good solution, and glslc doesn't work all that different from glslang, and uses it underneath.

• I can't get glslangValidator to recognize my imports (I don't just use the source location as a include directory), I don't know if it is VSCODE text parsing or what, but even with absolute file paths, I can't seem to get it to just use these other locations. It is definitely something on the VSCODE end of things, because I can manually enter the command line args just fine, this is what I tried to do:

• Doesn't work "glsllint.glslangValidatorArgs": "-I\"${env:TEST_WORKSPACE_ENV}/shaderutils/\" -V"

• This is what worked outside of vscode. $VULKAN_SDK/Bin/glslangValidator.exe -V -I"$TEST_WORKSPACE_ENV/shaderutils/" example.comp

• Our code however generates depfiles from GLSLC, which means it's actually possible to completely understand the dependency chain from build artifacts, instead of having users manually enter in each file. Currently we only use a couple locations for includes, so this isn't a huge deal for us (provided we can get around the include problems apparently specific to this extension and VSCODE), just a thought.

[hsimpson]

• Doesn't work "glsllint.glslangValidatorArgs": "-I\"${env:TEST_WORKSPACE_ENV}/shaderutils/\" -V"

Unfortunately "glsllint.glslangValidatorArgs" does not support ${env:YOUR_ENV} substitution (could possible another feature, see #48 )

Could you try to use a "full path" for testing?

[hsimpson]

@Cazadorro I close this issue here and release the testing version.