Blender 4.0 breaks blender_3dmigoto.py

[leotorrez]

Good afternoon, the recent update has removed vertex attributes in the form that are used in the plugin in favor of their attribute api. I Chat-GPT'd my way through the issue to try and find a solution myself and arrived to color_attributes.

here are the 3 functions I've updated with the suggested code. I don't comprehend in full what the tool is doing in all these steps hence why I didn't go for a pull request. But here are the 3 functions I had to update in order to make the export work once more.

# This loads unknown data from the vertex buffers as vertex layers
def import_vertex_layers(mesh, obj, vertex_layers):
    for (element_name, data) in sorted(vertex_layers.items()):
        dim = len(data[0])
        cmap = {0: 'x', 1: 'y', 2: 'z', 3: 'w'}
        for component in range(dim):

            if dim != 1 or element_name.find('.') == -1:
                layer_name = '%s.%s' % (element_name, cmap[component])
            else:
                layer_name = element_name

            if type(data[0][0]) == int:
                # Create an integer type custom vertex attribute
                layer = mesh.color_attributes.new(name=layer_name, type='INT')
                for v in mesh.vertices:
                    val = data[v.index][component]
                    # Set the integer value for each vertex
                    layer.data[v.index] = val
            elif type(data[0][0]) == float:
                # Create a float type custom vertex attribute
                layer = mesh.color_attributes.new(name=layer_name, type='FLOAT')
                for v in mesh.vertices:
                    # Set the float value for each vertex
                    layer.data[v.index] = data[v.index][component]
            else:
                raise ValueError('BUG: Bad layer type %s' % type(data[0][0]))
def blender_vertex_to_3dmigoto_vertex(mesh, obj, blender_loop_vertex, layout, texcoords, blender_vertex=None):
    if blender_loop_vertex is not None:
        blender_vertex = mesh.vertices[blender_loop_vertex.vertex_index]
    vertex = {}
    seen_offsets = set()

    # TODO: Warn if vertex is in too many vertex groups for this layout,
    # ignoring groups with weight=0.0
    vertex_groups = sorted(blender_vertex.groups, key=lambda x: x.weight, reverse=True)

    for elem in layout:
        if elem.InputSlotClass != 'per-vertex':
            continue

        if (elem.InputSlot, elem.AlignedByteOffset) in seen_offsets:
            continue
        seen_offsets.add((elem.InputSlot, elem.AlignedByteOffset))

        semantic_translations = layout.get_semantic_remap()
        translated_elem_name, translated_elem_index = \
                semantic_translations.get(elem.name, (elem.name, elem.SemanticIndex))

        # Some games don't follow the official DirectX UPPERCASE semantic naming convention:
        translated_elem_name = translated_elem_name.upper()
        if translated_elem_name == 'POSITION':
            vertex[elem.name] = list(blender_vertex.co)
            if 'POSITION.w' in mesh.color_attributes:
                w_attribute = mesh.color_attributes['POSITION.w']
                vertex[elem.name].append(w_attribute.data[blender_loop_vertex.vertex_index])
            else:
                vertex[elem.name] += [1.0]
            # Handle POSITION.w attribute if available
        elif translated_elem_name.startswith('COLOR'):
            color_attr_name = elem.name if elem.name in mesh.vertex_colors else elem.name + '.RGB'
            color_attr = mesh.color_attributes.get(color_attr_name)
            if color_attr:
                vertex[elem.name] = list(color_attr.data[blender_loop_vertex.index])
                if color_attr_name != elem.name:
                    alpha_attr = mesh.color_attributes[color_attr_name + '.A']
                    vertex[elem.name].append(alpha_attr.data[blender_loop_vertex.index][0])
        elif translated_elem_name == 'NORMAL':
            if 'NORMAL.w' in mesh.color_attributes:
                normal_w_attribute = mesh.color_attributes['NORMAL.w']
                vertex[elem.name] = list(blender_loop_vertex.normal) + \
                    [normal_w_attribute.data[blender_loop_vertex.vertex_index]]
            elif blender_loop_vertex:
                vertex[elem.name] = elem.pad(list(blender_loop_vertex.normal), 0.0)
            else:
                vertex[elem.name] = elem.pad(list(blender_vertex.normal), 0.0)
        elif translated_elem_name.startswith('TANGENT'):
            # DOAXVV has +1/-1 in the 4th component. Not positive what this is,
            # but guessing maybe the bitangent sign? Not even sure it is used...
            # FIXME: Other games
            if blender_loop_vertex:
                vertex[elem.name] = elem.pad(list(blender_loop_vertex.tangent), blender_loop_vertex.bitangent_sign)
            else:
                # XXX Blender doesn't save tangents outside of loops, so unless
                # we save these somewhere custom when importing they are
                # effectively lost. We could potentially calculate a tangent
                # from blender_vertex.normal, but there is probably little
                # point given that normal will also likely be garbage since it
                # wasn't imported from the mesh.
                pass
        elif translated_elem_name.startswith('BINORMAL'):
            # Some DOA6 meshes (skirts) use BINORMAL, but I'm not certain it is
            # actually the binormal. These meshes are weird though, since they
            # use 4 dimensional positions and normals, so they aren't something
            # we can really deal with at all. Therefore, the below is untested,
            # FIXME: So find a mesh where this is actually the binormal,
            # uncomment the below code and test.
            # normal = blender_loop_vertex.normal
            # tangent = blender_loop_vertex.tangent
            # binormal = numpy.cross(normal, tangent)
            # XXX: Does the binormal need to be normalised to a unit vector?
            # binormal = binormal / numpy.linalg.norm(binormal)
            # vertex[elem.name] = elem.pad(list(binormal), 0.0)
            pass
        elif translated_elem_name.startswith('BLENDINDICES'):
            i = translated_elem_index * 4
            vertex[elem.name] = elem.pad([ x.group for x in vertex_groups[i:i+4] ], 0)
        elif translated_elem_name.startswith('BLENDWEIGHT'):
            # TODO: Warn if vertex is in too many vertex groups for this layout
            i = translated_elem_index * 4
            vertex[elem.name] = elem.pad([ x.weight for x in vertex_groups[i:i+4] ], 0.0)
        elif translated_elem_name.startswith('TEXCOORD') and elem.is_float():
            # FIXME: Handle texcoords of other dimensions
            uvs = []
            for uv_name in ('%s.xy' % elem.remapped_name, '%s.zw' % elem.remapped_name):
                if uv_name in texcoords:
                    uvs += list(texcoords[uv_name][blender_loop_vertex.index])
            # Handle 1D + 3D TEXCOORDs. Order is important - 1D TEXCOORDs won't
            # match anything in above loop so only .x below, 3D TEXCOORDS will
            # have processed .xy part above, and .z part below
            for uv_name in ('%s.x' % elem.remapped_name, '%s.z' % elem.remapped_name):
                if uv_name in texcoords:
                    uvs += [texcoords[uv_name][blender_loop_vertex.index].x]
            vertex[elem.name] = uvs
        else:
            # Unhandled semantics are saved in vertex layers
            data = []
            for component in 'xyzw':
                layer_name = '%s.%s' % (elem.name, component)
                if layer_name in mesh.color_attributes:
                    data.append(mesh.color_attributes[layer_name].data[blender_loop_vertex.vertex_index])
            if data:
                vertex[elem.name] = data

        if elem.name not in vertex:
            print('NOTICE: Unhandled vertex element: %s' % elem.name)
        #else:
        #    print('%s: %s' % (elem.name, repr(vertex[elem.name])))

    return vertex
def blender_vertex_to_3dmigoto_vertex_outline(mesh, obj, blender_loop_vertex, layout, texcoords, export_Outline):
    blender_vertex = mesh.vertices[blender_loop_vertex.vertex_index]
    pos = list(blender_vertex.undeformed_co)
    blp_normal = list(blender_loop_vertex.normal)
    vertex = {}
    seen_offsets = set()

    # TODO: Warn if vertex is in too many vertex groups for this layout,
    # ignoring groups with weight=0.0
    vertex_groups = sorted(blender_vertex.groups, key=lambda x: x.weight, reverse=True)

    for elem in layout:
        if elem.InputSlotClass != 'per-vertex':
            continue

        if (elem.InputSlot, elem.AlignedByteOffset) in seen_offsets:
            continue
        seen_offsets.add((elem.InputSlot, elem.AlignedByteOffset))

        if elem.name == 'POSITION':
            pos = list(blender_vertex.co)

            if 'POSITION.w' in mesh.color_attributes:
                position_w_attr = mesh.color_attributes['POSITION.w']
                vertex[elem.name] = pos + [position_w_attr.data[blender_loop_vertex.vertex_index]]
            else:
                vertex[elem.name] = pos + [1.0]
             # Handling if 'POSITION.w' attribute is not found
        elif elem.name.startswith('COLOR'):
            if elem.name in mesh.vertex_colors:
                vertex[elem.name] = elem.clip(list(mesh.vertex_colors[elem.name].data[blender_loop_vertex.index].color))
            else:
                try:
                    vertex[elem.name] = list(mesh.vertex_colors[elem.name+'.RGB'].data[blender_loop_vertex.index].color)[:3] + \
                                            [mesh.vertex_colors[elem.name+'.A'].data[blender_loop_vertex.index].color[0]]
                except KeyError:
                    raise Fatal("ERROR: Unable to find COLOR property. Ensure the model you are exporting has a color attribute (of type Face Corner/Byte Color) called COLOR")
        elif elem.name == 'NORMAL':
            vertex[elem.name] = elem.pad(blp_normal, 0.0)
        elif elem.name.startswith('TANGENT'):
            vertex[elem.name] = elem.pad(export_Outline.get(blender_loop_vertex.vertex_index, blp_normal), blender_loop_vertex.bitangent_sign)
        elif elem.name.startswith('BINORMAL'):
            pass
        elif elem.name.startswith('BLENDINDICES'):
            i = elem.SemanticIndex * 4
            vertex[elem.name] = elem.pad([ x.group for x in vertex_groups[i:i+4] ], 0)
        elif elem.name.startswith('BLENDWEIGHT'):
            # TODO: Warn if vertex is in too many vertex groups for this layout
            i = elem.SemanticIndex * 4
            vertex[elem.name] = elem.pad([ x.weight for x in vertex_groups[i:i+4] ], 0.0)
        elif elem.name.startswith('TEXCOORD') and elem.is_float():
            # FIXME: Handle texcoords of other dimensions
            uvs = []
            for uv_name in ('%s.xy' % elem.name, '%s.zw' % elem.name):
                if uv_name in texcoords:
                    uvs += list(texcoords[uv_name][blender_loop_vertex.index])

            vertex[elem.name] = uvs
        else:
            # Unhandled semantics are saved in vertex layers
            data = []
            for component in 'xyzw':
                layer_name = '%s.%s' % (elem.name, component)
                if layer_name in mesh.color_attributes:
                    data.append(mesh.color_attributes[layer_name].data[blender_loop_vertex.vertex_index])
            if data:
                vertex[elem.name] = data

        if elem.name not in vertex:
            print('NOTICE: Unhandled vertex element: %s' % elem.name)

    return vertex

Apologies if this was already taking into consideration before the time of my post.

Blender 4.0 Patch Notes: https://wiki.blender.org/wiki/Reference/Release_Notes/4.0/Python_API

[DarkStarSword]

While it is impressive that ChatGPT was able to do this conversion, as as pretty typical for it, it has made a number of mistakes that I'll need to take a closer look at when I get back home:

• I'm not sure if color attributes will be a suitable replacement - Blender already has vertex color layers, but they have restrictions on what data can be placed into them (i.e. it has to be a colour), while the vertex layers are used to store any arbitrary data that either the script or Blender cannot handle. We already use color layers for COLOR attributes that meet the restrictions, vertex layers are for things that either aren't colours or are COLORs that don't meet those restrictions. It's possible that maybe color "attributes" might not be the same as color "layers", and perhaps the restrictions don't apply - but we need to confirm that. (Also, if they don't have the same restrictions we might be able to rework how we handle alpha channels in 4 component COLOR semantics)
• The vertex layers need to be abstracted so the script will continue working in Blender 3.x (or alternatively we need to fork it for 4.0, but I am very much against doing so unless the differences become too extreme)
• It's deleted the handling of large unsigned integers, unless the color layers are unsigned this will break some games
• It's made an assumption that elem.pad will always pad the POSITION to 4 components and replaced the function that handles this properly with a naive implementation that always pads it, but this is not the case where the POSITION semantic was only 3 components
• Need to examine exactly what it's changed with COLOR attributes - the changes here are non-trivial and I'm not convinced they are correct or even sensible
• It's deleted several comments that were unrelated to it's changes, were not out of date, and that it does not otherwise address.
• blender_vertex_to_3dmigotovertexoutline is new, but never called (I haven't reviewed the contents of this function yet)

I also need to check what happens to the vertex layers if Blender 4.0 opens a Blender 3.x blend file

[AkaShrug]

seems like you can define custom attribute ? have not looked much at it

[DarkStarSword]

Yeah, I think this is what we should be using, the point domain should be fairly equivalent to vertex layers - looks like it was added in 3.0, so there are a number of versions where either attributes or vertex layers could be used: https://docs.blender.org/manual/en/latest/modeling/geometry_nodes/attributes_reference.html

[DarkStarSword]

I've pushed up a change to use custom attributes on 4.x, while retaining backwards compatibility with Blender 3.x vertex layers.

This has only been lightly tested, so please let me know if you encounter any issues with them.

[DarkStarSword]

Surprisingly migrating a .blend file saved with Blender 3.x to Blender 4.x appears that it might "just work", because in Blender 3.x vertex layers were already showing up in the mesh attributes, and since I haven't changed the names the script will just find them there when exporting in 4.x. That said, if anyone encounters issues doing this, it might be worth just re-importing the mesh, either from raw buffers exported with 3.x, or a frame analysis dump.