Blender DNA structs -> C#

[McManning]

Just a place for me to dump notes on this topic -

Depending on how much support I'm going to need and performance I need to squeeze out of this where I can't just do structs -> python -> C#, I'll need to have a realistic solution for just interacting with Blender DNA directly.

The primary problem is DNA struct changes between Blender versions, and having to build a DLL per-version or some technique in order to extract data I need without worrying too much about version changes.

Current / known use cases

1. Fast access to vertex weight data Mesh->dvert which currently requires an obscene number of memcpy ops if through Python
2. Fast access to custom split normal data Mesh->ldata which currently does not seem easily accessible via Python in bulk - requiring a per-vertex-copy operation which is prohibitively slow.

Both of the above are on the Mesh struct - and unfortunately the top of the Mesh struct is an ID which seems to be changing in size each version. Already, I'd need to have a copy of ID defined in C# before even defining Mesh.

3. Fast access to grease pencil strokes and point data bGPDstroke/bGPDspoint/etc

Considering how active GP development is, I expect these structs to change frequently.

Solutions

Autogenerated definitions

Overengineered solution - automatically generate a C# definitions file that includes all the Blender structs I need and their changes between different Blender versions.

Using a modified version of something like CppHeaderToCSharpConverter it could run through checkout revisions of Blender that we support, pull struct files, parse, and then create namespaced/grouped blocks of code representing each extracted struct that we need. As the DLL is loaded, Python passes the current Blender version in and the appropriate structs are used for all data extraction.

E.g. something like:

// Autogenerated structs.cs
namespace BlenderDNA.v292 {
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public struct ID {
      ..
    }

    public struct Mesh {
      ..
    }
}

[StructLayout(LayoutKind.Sequential, Pack = 1)]
namespace BlenderDNA.v291 {
    ..
}

This, of course, is horrifying.

Python byte skipping

If I could figure out what offset Mesh->ldata is from the initial ptr of Mesh in Python - I could potentially do a .as_pointer() for the mesh, and then offset from that. Or - if I could grab an initial entry in an array like I do for MVert/MLoopTri/etc.

The main problem - each piece of data I'm grabbing needs a custom solution. dvert can't be grabbed from Python (only the underlying vertex weights that are referenced by each dvert entry can).

[McManning]

Some references -

LuxCore does something similar as they also need split normals. Their approach is struct-per-version which looks to be a pain in the ass to maintain for future versions - https://github.com/LuxCoreRender/LuxCore/blob/master/src/luxcore/pyluxcoreforblender.cpp#L719-L727

For weights, I can get access to the weight list through Python but not the array of weight lists for all vertices - so I'm still iterating vertices in Python and calling into Blender to get the pointer to each weight group. Reference: https://github.com/sobotka/blender/blob/af316d276144b905cf6cf5a1b1d7ae727d545e2f/source/blender/makesrna/intern/rna_mesh.c#L527-L531

[McManning]

Maybe something like https://github.com/WaveEngine/RenderDoc.NET

Under the hood it uses https://github.com/xoofx/CppAst.NET for parsing C headers

[McManning]

Implemented as https://github.com/McManning/SharpRNA

Will now just need to integrate back into this project.

[McManning]

Obsolete with the new C++ core