Avoid automatic vertex inputs reorder

[pratamabayu]

Hi @floooh ,

I got issue about vertex inputs order. I have shader like

layout (location = POSITION) in vec4 pos;
layout (location = TEXCOORD0) in vec2 texcoord0;
layout (location = BLENDINDICES) in vec4 boneIndices;
layout (location = BLENDWEIGHT) in vec4 boneWeights;

but when translate to other shader, like GLSL, vertex inputs reordered automatically to be

layout(location = 16) in vec4 boneIndices;
layout(location = 17) in vec4 boneWeights;
layout(location = 0) in vec4 pos;
layout(location = 2) in vec2 texcoord0;

How to avoid it?

Thank you,

[floooh]

What values are POSITION, TEXCOORD0, BLENDINDICES and BLENDWEIGHT?

In general, the order how anything appears after the retranslation is outside my control (in this case it looks like one of the translation steps orders alphabetically, most likely SPIRVCross), but the order also shouldn't matter as long as the layout(location=...) remain unchanged from the input GLSL.

I don't think that sokol_gfx.h accepts gaps in the vertex attribute locations btw, are you using something else for rendering?

[pratamabayu]

What values are POSITION, TEXCOORD0, BLENDINDICES and BLENDWEIGHT?

Vector4 for POSITION, BLENDINDICES, and BLENDWEIGHT; and Vector2 for TEXCOORD0

In general, the order how anything appears after the retranslation is outside my control (in this case it looks like one of the translation steps orders alphabetically, most likely SPIRVCross)

Ya, It seems internally in SPIRVCross. Maybe can be done if SPIRVCross has API to reorder resources.stage_inputs().

but the order also shouldn't matter as long as the layout(location=...) remain unchanged from the input GLSL

I need predictable order based on original shader (before translation using SPIRVCross) when creating VertexBuffer. This's how I created the VertexBuffer if order is predictable.

VertexBufferHandle createVertexBuffer(const Memory& t_memory)
        {
            sg_buffer_desc desc = { };

            if (t_memory.data)
                desc.data = { t_memory.data, t_memory.size };
            else {

                desc.usage = SG_USAGE_DYNAMIC;
                desc.size = t_memory.size;
            }

            sg_buffer buffer = sg_make_buffer(&desc);

            return { buffer.id };
        }

void Mesh::invalidateChanges()
    {
        ...

        VertexFlags flags = VertexFlags::Position;

        if (m_normals.size() > 0)
            flags = flags | VertexFlags::Normal;
        if (m_textureCoordinate0.size() > 0)
            flags = flags | VertexFlags::TextureCoordinate0;
        if (m_textureCoordinate1.size() > 0)
            flags = flags | VertexFlags::TextureCoordinate1;
        if (m_textureCoordinate2.size() > 0)
            flags = flags | VertexFlags::TextureCoordinate2;
        if (m_textureCoordinate3.size() > 0)
            flags = flags | VertexFlags::TextureCoordinate3;
        if (m_colors.size() > 0)
            flags = flags | VertexFlags::Color0;
        if (m_tangents.size() > 0)
            flags = flags | VertexFlags::Tangent;
        if (m_boneWeights.size() > 0)
            flags = flags | VertexFlags::BlendWeight | VertexFlags::BlendIndices;

        BinaryWriter writer;

        for (int i = 0; i < m_vertices.size(); ++i)
        {
            if ((flags & VertexFlags::Position) == VertexFlags::Position)
            {
                writer.write(m_vertices[i].x);
                writer.write(m_vertices[i].y);
                writer.write(m_vertices[i].z);
            }

            if ((flags & VertexFlags::Normal) == VertexFlags::Normal)
            {
                writer.write(m_normals[i].x);
                writer.write(m_normals[i].y);
                writer.write(m_normals[i].z);
            }

            if ((flags & VertexFlags::TextureCoordinate0) == VertexFlags::TextureCoordinate0)
            {
                writer.write(m_textureCoordinate0[i].x);
                writer.write(m_textureCoordinate0[i].y);
            }

            if ((flags & VertexFlags::TextureCoordinate1) == VertexFlags::TextureCoordinate1)
            {
                writer.write(m_textureCoordinate1[i].x);
                writer.write(m_textureCoordinate1[i].y);
            }

            if ((flags & VertexFlags::TextureCoordinate2) == VertexFlags::TextureCoordinate2)
            {
                writer.write(m_textureCoordinate2[i].x);
                writer.write(m_textureCoordinate2[i].y);
            }

            if ((flags & VertexFlags::TextureCoordinate3) == VertexFlags::TextureCoordinate3)
            {
                writer.write(m_textureCoordinate3[i].x);
                writer.write(m_textureCoordinate3[i].y);
            }

            if ((flags & VertexFlags::Color0) == VertexFlags::Color0)
            {
                auto color = m_colors[i].toVector4();

                writer.write(color.x);
                writer.write(color.y);
                writer.write(color.z);
                writer.write(color.w);
            }

            if ((flags & VertexFlags::Tangent) == VertexFlags::Tangent)
            {
                writer.write(m_tangents[i].x);
                writer.write(m_tangents[i].y);
                writer.write(m_tangents[i].z);
                writer.write(m_tangents[i].w);
            }

            if ((flags & VertexFlags::BlendIndices) == VertexFlags::BlendIndices)
            {
                writer.write(m_boneWeights[i].boneIndex0);
                writer.write(m_boneWeights[i].boneIndex1);
                writer.write(m_boneWeights[i].boneIndex2);
                writer.write(m_boneWeights[i].boneIndex3);
            }

            if ((flags & VertexFlags::BlendWeight) == VertexFlags::BlendWeight)
            {
                writer.write(m_boneWeights[i].weight0);
                writer.write(m_boneWeights[i].weight1);
                writer.write(m_boneWeights[i].weight2);
                writer.write(m_boneWeights[i].weight3);
            }
        }

        m_vertexBufferHandle = RenderEngine::createVertexBuffer({ writer.data(), writer.size() }).id;

        writer.destroyBuffer();

        ...
    }

I don't think that sokol_gfx.h accepts gaps in the vertex attribute locations btw, are you using something else for rendering?

I use sokol_gfx.h, sokol_app.h, sokol_audio, and sokol_fetch.h for my little game engine.

[floooh]

Vector4 for POSITION, BLENDINDICES, and BLENDWEIGHT; and Vector2 for TEXCOORD0

Not the types, but the values, e.g. I guess that POSITION=0, TEXCOORD0=2, BLENDINDICES=16 and BLENDWEIGHTS=17 (btw, this might be another problem, sokol_gfx.h only allows up to 16 vertex attribute slots, this is a limititation from underlying 3D APIs).

But apart from that, are you aware of the sg_vertex_attrib_state.offset member? This allows to decouple the vertex attribute order from the order of vertex-components in the mesh data, e.g. you could do something like:

sg_make_pipeline(&(sg_pipeline_desc){
    .layout = {
        [0] = { .offset = 0, .format = SG_VERTEXFORMAT_FLOAT3,
        [1] = { .offset = 24, .format = SG_VERTEXFORMAT_FLOAT2,
        [2] = { .offset = 12, .format = SG_VERTEXFORMAT_FLOST3,
    },
});

Note how the vertex data for vertex attribute in slot [1] is located after the vertex data for the attribute in slot [2].

This also allows to generate a single mesh with all vertex components for all shaders and pipeline objects, and let specific pipeline objects and their shaders use a subset of vertex components in the shared mesh.

Finally sokol-shdc has an optional reflection output (activate via --reflection) which let's you query all sorts of information (for instance vertex attribute slots by name), see this example:

https://floooh.github.io/sokol-html5/shdfeatures-sapp.html

C source:

https://github.com/floooh/sokol-samples/blob/master/sapp/shdfeatures-sapp.c

Shader source:

https://github.com/floooh/sokol-samples/blob/master/sapp/shdfeatures-sapp.glsl

The sokol_shapes.h header is similar, it always generate ALL vertex data (position, normal, texcoords and color), but when rendering you are free to use only a subset of those vertex components. For instance here I'm creating a pipeline object which only uses the position and normal (e.g. I'm ignoring texcoord and color):

https://github.com/floooh/sokol-samples/blob/3e746d86cd87f634f4f7793c2e8d0ef71a2067ca/sapp/offscreen-msaa-sapp.c#L146-L152

...and a bit below position, normal and texcoord0 (and ignore the color) from the same mesh data:

https://github.com/floooh/sokol-samples/blob/3e746d86cd87f634f4f7793c2e8d0ef71a2067ca/sapp/offscreen-msaa-sapp.c#L169-L174

I use sokol_gfx.h ...

Strange, is the validation layer enabled? Some of the things you are trying to do sound like they should trigger validation errors...

[floooh]

PPS: regarding this:

I need predictable order based on original shader

This I have trouble to understand... are you somehow parsing the output shader code looking for vertex attribute definitions? That's the only way I can think of how the vertex attribute order in the generated GLSL shader file would matter.

[pratamabayu]

Sorry for late reply,

I was investigate it. The problem is stage input reordered by usage in main function.

Source example.

...
layout (location = POSITION) in vec4 position;
layout (location = TEXCOORD0) in vec2 texcoord0;
layout (location = BLENDINDICES) in vec4 boneIndices;
layout (location = BLENDWEIGHT) in vec4 boneWeights;
...
void main() {

    // apply skinning
    mat4 skinMat = 
        bones[int(boneIndices.x)] * boneWeights.x +
        bones[int(boneIndices.y)] * boneWeights.y +
        bones[int(boneIndices.z)] * boneWeights.z +
        bones[int(boneIndices.w)] * boneWeights.w;

    vec4 skinPosition = skinMat * position;

    gl_Position = modelViewProjection * skinPosition;
    __texcoord0 = texcoord0;
}

It will produce stage inputs like below.

...
layout (location = ...) in vec4 boneIndices;
layout (location = ...) in vec4 boneWeights;
layout (location = ...) in vec4 position;
layout (location = ...) in vec2 texcoord0;
...

So, i do pre-process the shader to create some temporary variables in main function automatically based on stage inputs order before glslang processing.

void main() {
vec4 __temp_position = position;
vec4 __temp_texcoord0 = texcoord0;
vec4 __temp_boneIndices = boneIndices;
vec4 __temp_boneWeights = boneWeights;
...
}

Then I leave it to the optimization section in spirv-cross to remove the temporary variables. If anyone is interested, here is a function to modify the source.

void fixStageInputsOrders(std::string& t_source) {

            std::vector<VertexInput> inputs;
            std::istringstream stream(t_source);

            std::string line;
            while (std::getline(stream, line)) {

                // regular expression pattern to match "layout (location = POSITION) in vec4 position;"
                static std::regex pattern(R"(layout\s*\(\s*location\s*=\s*\w+\s*\)\s*(in)\s*(\w+)\s*(\w+)\s*;)");

                // match results
                std::smatch matches;

                // attempt to match the input string with the regular expression pattern
                if (std::regex_match(line, matches, pattern)) {

                    // extract the "in", "vec4", and "position" parts
                    std::string qualifier = matches[1].str();
                    std::string type = matches[2].str();
                    std::string variableName = matches[3].str();

                    // put the extracted parts
                    inputs.push_back({ type, variableName });
                }
            }

            std::string additionalLines;
            for (const auto& input : inputs) {

                additionalLines += input.type + " __temp_" + input.name + " = " + input.name + ";\n";
            }

            static std::regex pattern("void\\s+main\\s*\\(\\s*\\)\\s*\\{");

            std::smatch match;
            if (std::regex_search(t_source, match, pattern)) {

                // get the position of the match
                size_t mainFuncPos = match.position();

                // insert your additional lines after "void main() {"
                t_source.insert(mainFuncPos + match.length(), additionalLines);
            }
        }

I need this solution because my Mesh design pattern need to know the original stage input orders.

Thank you @floooh