[Questions] Some questions regarding correct procedure for reading

I'm working on a tool which converts a scene format from a game to glTF, and then can convert glTF back to that scene format.

The first part is working great though I am having some trouble implementing the latter half - mainly because I'm not entirely sure what the correct way to get all the information I need from the glTF is.

For the "conversion to glTF" portion I've been taking advantage of the Toolkit classes such as SceneBuilder, MeshBuilder, etc. and have had great success.

However I feel a bit in the dark when doing the reverse and I was looking for some general advice and assistance with this.

Since the Toolkit API is what I'm familiar with, my instinct is to SceneBuilder.LoadDefaultScene the glTF file. From there, I see that the only thing I seem to have to access the scene's contents is the Instances property, and so I begin iterating through the InstanceBuilders in that.

It then seems that the only way to get anything useable from there is ContentTransformer via the InstanceBuilders' Content properly, and from there use the Get[X]Asset functions to try and get different types of assets to try and determine what kind of asset is being worked with.

Is all of this correct or am I going about this in the complete wrong way? I'm not entirely sure because it feels like I'm doing something wrong.

There's also the issue of the mesh's armature - because of how NodeBuilder works I have no idea how to actually get the full skeleton properly. ContentTransformer.GetArmatureRoot() will get me the root "Armature" node. That's good, but in the exported file all the meshes are child nodes of the armature. And since NodeBuilder is generic, attempting to iterate through the armature to build the model's skeleton means that I end up including all the mesh nodes in that too which is obviously not what I want. But I can't seem to see how I'm intended to distinguish what is what when working with NodeBuilders?

I'd love some general advice on all this because documentation on this aspect of SharpGLTF seems a bit tricky to come by.

Some prior notes:

glTF is a very complex format, the reason the toolkit exists is to ease the burden of writing models.

Reading models is a totally different scenario, and depending on what you want to do, there's different approaches. Also, these approaches are very different depending on whether you are converting rigid or skinned models, given you mention GetArmatureRoot, I take it it's skinned models.

The armature root is this: Originally, glTF had a mandatory property called "skeleton" to define the skeleton root, but since it can be effectively computed from the data, the glTF board made it optional, and recomended to write it to the file for backwards compatibility, but there's no requirement of a model to have the "skeleton root" being defined, and since when loading a model I always take the worst case scenario (not defined), that's why you can't really find the skeleton root.

Basically, the skeleton root is: given a collection of skinned joints(nodes), find the common parent of all of them, and that will be the skeleton root node. Notice that this node may be part of the skin, or may be outside the skin joint collection.

Reading a model for the purpose of rendering, or exporting, I call it "model evaluation", because glTF is essentially a state machine.

If the Toolkit is a helper library designed to write models, there's also the SharpGLTF.Runtime library, which is a helper class designed to read a model with the aim of rendering it , this detail is important because the runtime library evaluates the scene under the hood, and tells a graphics engine exactly where to render each mesh and how.
In the toolkit there's also an extension called Scene.EvaluateTriangles, this extension basically outputs a plain collection of triangles that is a snapshot of the current state of the model at a given time (if it's animated), and these methods, internally, do all the work of evaluating the node transforms, animations, skins, etc, to deliver the final triangles.
Finally there's also a method somewhere (there's unit tests using it) to convert a default scene to a SceneBuilder, so you might want to use that approach if you're familiar with the toolkit. This path of converting a scene back to a SceneBuilder is originally intended for use cases when you need to modify the source scene in some way. But for pure export, I would personally use one of the other approaches, because when converting scene=>scenebuilder some data might be lost, so only use this approach if you fail to use the previous two approaches.

There's also the problem that when converting glTF to some other format, the glTF may have more features than the target format; for example, glTF supports multiple skin objects in a single scene, whereas many game formats only support one skin object, so not all glTFs may be able to be exported to a given target format

So I actually already have some of the basics working, it is mainly this whole issue with determining node types and rebuilding the skeleton for the game that's the main issue.

Here's an example why: In Blender, the meshes for an armature are children of that armature, like this:

When calling GetArmatureRoot, it returns the main "Armature" object, which is somewhat expected. However my expectation would be able to able to recursively iterate through the skeleton NodeBuilder hierarchy from there.

However, the problem lies in the fact that any meshes are nodes part of the Armature's hierarchy. This means that when I try to iterate through the skeleton, I end up including meshes in it as well. Normally I'd find some way to just filter them out and skip them, but when using NodeBuilders it doesn't seem possible to distinguish which of these elements are actually the bones.

Regarding the other methods - EvaluateTriangles doesn't seem useful to me because to properly convert the model I need all the information rather than the "evaluated" version of the mesh. I'm also not sure about the Runtime library, since my goal is to convert from glTF into a different format, but I'm not entirely sure because I only took a precursory glance at how it works.

I do see that the Schema2 objects have several properties such as IsSkinSkeleton and IsSkinJoint which seems to be exactly what I need but I don't think there's an easy way to correlate NodeBuilders to their corresponding Schema2 object. Will I just have to rewrite my code to iterate through the Schema2 objects instead of using the Toolkit classes?

I think the problem is you're expecting a high level structure that glTF does not have in the way you wan.

Consider that the original purpose of glTF was to distill all the high level constructs and concepts of authoring 3D models into a series of low level objects that could be easily chewed by a graphics engine for rendering.

So glTF does not have concepts like "armatures" or "bones" or "skeletons", it's more low level than that.

You need to understand how to "evaluate" a glTF scene, and I guess you'll have to do some homework on that. glTF.Runtime library is a good example of how to "evaluate" a glTF scene, but for a brief explanation, it boils down to this:

You traverse all the nodes of a scene
- For every node that has a Mesh reference, you render that (rigid) mesh in the world transform location of that node.
Create a Lookup Table of all nodes of the scene, and its world positions at a given time
For every Skin in the scene (inside the nodes), you feed the joints from the previously generated lookup table, and you render the skinned model using these joints - notice that the transform of the node containing the skin does not play any role in the rendering of the skinned model, and the node is used just as a mere container of the skin.

Now, I know what you want is to export the glTF to another format, but I think it's important for you to understand how a glTF would be rendered so you can understand how to export it. Because glTF has a flexibility that most probably your target format does not support. For example, glTF supports multiple skinned meshes within the same scene, and most probably your target format only supports one.

If that would be the case, I would ensure that the imported glTF has one, and only one skin object, otherwise you throw an error, then, you pick the skin, and figure out all the nodes that would be required to fill the joints needed by the skin, and that would be your skeleton.

I can't explain it better in less words... skinning is a complex topic that requires a full course.

IsSkinJoint just tells that this particular node is being referenced by at least one skin object (a node can be reference by multiple skin objects)

If I had to do an exporter, yes, I would use the Schema2 objects to process the scene, the skin, the skeletons, etc. The meshes you could try convert them to MeshBuilder if you're more comfortable with them. But the scene..... the scene you really need to understand it... not the library, but the glTF architecture and design concepts.

The target format is a scene format - thus it supports multiple scenes, domains within those scenes, multiple meshes, skinned or not, within those domains. It's clearly not as low level at glTF but it has plenty of concepts that translate well, which factored into the choice to use glTF.

I did end up solving my problem, though the solution is definitely something of a hack. While the target format supports multiple skeletons per-mesh, this is an incredibly uncommon use case and only 3-4 models in the game actually utilise this feature (models used for the protagonist).

As such, since updating my code to support multiple skeletons is such a low priority, I was able to find the one skeleton to convert by: Getting the armature root Iterating all logical nodes that are children of the armature root Finding the first node which has IsSkinJoint set - this is considered to be the root node of the skeleton and from there I can traverse that tree and convert it to the game's skeletal format.

Notice that creating the skeleton based on IsSkinJoint or the Skin object can be misleading.

The nodes that have IsSkinJoint enabled, which are the Nodes being used as joints by the Skin object, are the nodes that contribute their world transform directly into the mesh.

But consider that the world transforms of a node are calculated upon their respective parent nodes, even if these nodes are not part of the skinning. In this case, these nodes that are not in the skin, are indirect contributors.

I always give this example for people to understand the difference between a skinned mesh and a skeleton:

rayman

the rayman character does not have "elbows" nor "knees", so the skin object does not have these joints because the mesh does not have them either. Does it mean you don't need elbows nor knees in the skeleton? no, you still need the whole skeleton, even if parts of the mesh don't use them directly, because you need the whole skeleton in order to calculate the world transforms of the nodes that are used by the skin. Again, complex topic, hard to explain with few words

Then it seems to me that the simplest solution is that I'll have to either rely on the expectation that the root joint will have some influence on some part of the mesh, or use an Extras property to mark out which node serves as the root joint of the skeleton.

vpenades / SharpGLTF

[Questions] Some questions regarding correct procedure for reading #230