Getting Animation Frame Times (Sampler Input)

[ptasev]

I'm working on converting gltf to a custom game format using your library. Is there an easy way to get all the frame times of an animation? I'll need to be able to do that both for the entire animation regardless of which node it is for, and also per node as well. Once I have that, I can get snapshots of the data at the exact frame times that the artist intended.

[vpenades]

You need to export the animations, or you just need to take geometry snapshots at specific times?

Because if it's the latter, there's an API in the toolkit that already does that, in fact, you can save a geometry snapshot at a given time, to wavefront OBJ:

using SharpGLTF;
using SharpGLTF.Schema2;

var model = ModelRoot.Load("model.gltf");
var animation = model.LogicalAnimations[0];
var time = 0.53f;

model.SaveAsWavefront("model.obj", animation , time );

Internally, it uses the EvaluateTriangles method , that gives you a raw list of triangles representing the state of the scene at a given animation, and a given time.

If you want to access the animation channels directly, there's a number of APIs to do so:

using SharpGLTF;
using SharpGLTF.Schema2;

var model = ModelRoot.Load("model.gltf");

// we will retrieve the curves from Node 5, Animation 0
var animation = model.LogicalAnimations[0];
var node = model.LogicalNodes[5];

var Tsampler = animation.FindTranslationSampler(node);
var Rsampler = animation.FindRotationSampler(node);

var TranslationKeyValues = Tsampler.GetLinearKeys().ToArray();

Both Tsampler and Rsampler are of type IAnimationSampler<T> that can give you both the keys and the values, in linear and cubic sampling modes.

[ptasev]

I can already get the geometry data at a given time using your runtime API. Specifically, I use a SceneInstance, and SetAnimationFrame, then loop through the drawable references, and apply the IGeometryTransforms to the VertexData using functions like TransformPosition.

Thanks for the EvaluateTriangles reference. It should help me with getting the geometry at each time.

What I need is to get all of the frame times from an Animation so that I can evaluate the geometry at those times. The format I'm working with uses morph targets as the animation method, but it does so for the entire mesh. So I need to take a snapshot at each time in the animation, and record that time.

[vpenades]

well... if you want to sample the animation at specific intervals, I would consider this:

model.LogicalAnimations[0].Duration will give you the total time of the animation, so you just need to sample from 0 to Duration.

IAnimationSampler has a method called CreateCurveSampler that will give you a sampler that is able to give you a value at any time, not only at the keys.

In fact, EvaluateTriangles is already using it and it is able to do proper interpolation, so you don't need to worry about specific keys, you just need to picks samples at fixed intervals.

But if you still need the specific time keys, I would do something like this:

• For every animation in AnimationLogic, try to get all the animation IAnimationSampler for all the nodes.
• For every IAnimationSampler you collect (from T, R and S), check the interpolation mode and use GetLinearKeys or GetCubicKeys to get the key-value pairs.
• Do a Select(item => item1) to get only the key
• Agregate all the enumerables with a .Distinct() or add them to a HashSet();

The final collection will have all the keys used by any node on that animation.

Final note: I would not worry about the time keys, and do a sampling at fixed intervals from 0 to .Duration. The reason is that glTF supports cubic spline interpolation, so it is possible you get a model that uses cubic spline curves, which can use very few keys, but the values in between are anything but linear. So if you're going to use morph animation, you would miss these.

So sampling at fixed intervals is the way to go to convert from skeletal animation to morphing animation.

[ptasev]

I see what you're saying. Maybe I can have a setting for the sample rate in fps instead. I'll try it out and see how it goes.

I've been looking into the EvaluateTriangles method and it seems like it will be very useful. However, I'm currently stuck. My idea was to use model.DefaultScene.EvalutateTriangles(), but there are nodes with a specific name that I do not want to be part of the geometry data.

Is there any way to ignore these nodes? One of my thoughts was to simply create a new scene, and reference all the same nodes from the DefaultScene, except for the ones I want to ignore. I haven't been able to figure out which calls to make to do that. Is this possible?

[vpenades]

Hmm... that is a very specific request...

Under the hood, EvaluateTriangles uses the Runtime API to evaluate the scene,

Here's the code inside EvaluateTriangles, and where would I look into adding additional filtering.

var instance = Runtime.SceneTemplate
                .Create(scene, false)
                .CreateInstance();

            if (animation == null)
            {
                instance.SetPoseTransforms();
            }
            else
            {
                instance.SetAnimationFrame(animation.Name, time);
            }

            var meshes = scene.LogicalParent.LogicalMeshes;

            return instance
                .DrawableReferences
                . // in here you can probably filter the nodes
                .Where(item => item.Transform.Visible)
                .SelectMany(item => meshes[item.MeshIndex].EvaluateTriangles(item.Transform));

I would also remove the '.Where(item => item.Transform.Visible)' because you need all the frames to have the same number of vertices.

I understand that sampling at fixed framerate can use a lot of geometry..... mayve a solution would be to have an heuristic that scores how much one specific snapshot differs from another snapshot... so when you export, you advance the animation at a very short fixed step, but you only export one snapshot when the threshold signals that the old and new snapshots are different enough

[ptasev]

Using a heuristic is an interesting idea. It might increase the processing time by a lot to get the score, but certainly better than having a lot of geometry at run time. Maybe a similar idea could be to go back to getting all the keys and values of the animation and doing some sort of heuristic there. All good things to try, thanks!

I tried going down the route of adding an extension to scene to filter the nodes as well yesterday. The problem with filtering at DrawableReferences is that it only exposes the logical mesh index, and I'm pretty sure that multiple nodes can can use the same mesh so I can't look up the node names based on the mesh. I'll probably look deeper into the code, and see if I can expose the node index in DrawableReferences. Should I do a pull request to add NodeIndex?

[vpenades]

I would preffer that DrawableReferences to not have any direct reference to the source gltf Model. It is designed so when you create the template, you can let the GC to completely flush all the source model. And it has some work behind so it is future proof.

I will look into the problem, though, I might actually copy the name to the template too, let me analyze the problem in the next days.

In the meantime, there's a nasty shortcut you might want to try: DrawableReferences are built as you traverse the node tree and nodes with meshes are found. So if you have 5 nodes with meshes, you'll get 5 drawable references.

If you traverse the node tree in the same way as the template constructor does, you can find which drawable references to skip.

It's quite a lot of work, so you might preffer to wait for me to come up with a solution, and focus on other stuff for now.

[ptasev]

It seems the workflow for exporters are somewhat different from the workflow for runtime graphics. Looking forward to your solution!

By the way, while you're looking into that, I thought I'd bring up another point. I'm using EvaluateTriangles, and adding those to a mesh builder with a VertexColor1Texture1 vertex material. If the primitive does not have colors or texcoords there's no way to tell from the vertex material since it will fill the texcoords and colors with 0s if they don't exist. My format allows the colors and tex coords to be excluded, but since I have no way to tell with the API, I have to include them no matter what. I looked through the code, but couldn't find a good way to check if the primitive has colors or texcoords.

The only workaround I can think of is checking all the triangles in a primitive to see if all colors or all texcoords are 0. Maybe in the future this can be a property on PrimitiveBuilder (HasColors, HasTexCoords) that gets set internally in the "AddTriangle" method?

My workaround:

// Check if MeshBuilder has texcoords and colors
bool hasTexCoords = gltfMesh.Primitives.Any(p => p.Vertices.Any(v => v.Material.TexCoord != Vector2.Zero));
bool hasColors = gltfMesh.Primitives.Any(p => p.Vertices.Any(v => v.Material.Color != Vector4.Zero));

[vpenades]

About EvaluateTriangles, notice there's two variants: a generic one that requires you to specify a particular vertex type, and another one that returns IVertexBuilder instances for every vertex.

The IVertexBuilder does tell you (per vertex) what's included in the vertex, either with a Try method that will return false if it's not supported, or with Max properties that will tell you the amount of elements.

[vpenades]

Hi, in SceneInstance, I've added a GetDrawableInstance, and deprecates GetDrawableReference.

With this one, you can get the node name. Let me know if it works for you.

[ptasev]

I tested those changes, and it seems to work great, thank you! Looking forward to the next release!

While node name works for me, maybe it'll be a good idea to put LogicalNodeIndex instead of NodeName for the sake of future proofing DrawableInstance? Then you can use instance.LogicalNodes to access the node and name. I don't understand the code as well as you so I'm not sure if it's a good idea.

I have just one more problem left to take care of. For some reason I'm getting a validation error with the following gltf with both the jpg, and png version. (First I try one, then I open and edit the gltf file to try the other) I dropped them on the GLTF validator from Khronos here, and it said that there's no issue.

GltfTexturesGiveValidationError.zip

[vpenades]

I wanted to avoid using a LogicalNodeIndex because it implies that you must hold a reference to the source document, which is something I want to avoid as much as possible.

I'll look at the issue you mentioned, but I'm pretty sure it has something to do with white spaces in the file names.

[ptasev]

I tried your fix, and it works now for files with spaces, but Blender already escapes the string when it writes the uri. The library has a problem if it's already escaped and has %20 for spaces.

We can use Uri.UnescapeDataString, but we have to guarantee that the string is already escaped somehow. For now the best I could come up with is the following:

public static ReadContext CreateFromDirectory(string directoryPath)
{
    return new ReadContext(assetFileName =>
    {
        bool fileExists = File.Exists(Path.Combine(directoryPath, assetFileName));

        // If the file doesn't exist, try unescaping first
        if (!fileExists)
            assetFileName = Uri.UnescapeDataString(assetFileName);

        return new BYTES(File.ReadAllBytes(Path.Combine(directoryPath, assetFileName)));
    });
}

On the other hand maybe it's best to add this check when seeing if it needs to be escaped in the Guard.IsValidURI function:

// Attempt to fix URI if not already well formed
if (!Uri.IsWellFormedUriString(gltfURI, UriKind.RelativeOrAbsolute))
    gltfURI = Uri.EscapeUriString(gltfURI);

[vpenades]

Hi, I'll investigate whether gltf should support escaped strings or not... because if not, it's blender exporter's issue,

[vpenades]

Regarding the issue with invalid file names, I wrote a question here: https://github.com/KhronosGroup/glTF/issues/1449#issuecomment-611560284

And it's confirmed that file paths must be unescaped, I'll update the source code accordingly.

[ptasev]

Good to hear!

Thanks for all your support, and work into this project! It has saved me a lot of time!

[amit11996]

using System; using System.IO; using System.Linq; using System.Windows; using System.Windows.Media; using System.Windows.Media.Imaging; using System.Windows.Media.Media3D; using Microsoft.Win32; using SharpGLTF.Schema2; using HelixToolkit.Wpf; using System.Collections.Generic;

namespace GLBModelViewer { public partial class MainWindow : Window { private ModelVisual3D _modelVisual; private Model3DGroup _modelGroup;

    public MainWindow()
    {
        InitializeComponent();
    }

    private void LoadModelButton_Click(object sender, RoutedEventArgs e)
    {
        OpenFileDialog openFileDialog = new OpenFileDialog
        {
            Filter = "glb files (*.glb)|*.glb",
            Title = "Select a GLB Model"
        };

        if (openFileDialog.ShowDialog() == true)
        {
            LoadGLBModel(openFileDialog.FileName);
        }
    }

    private void LoadGLBModel(string filename)
    {
        try
        {
            ModelRoot gltfModel = ModelRoot.Load(filename);
            _modelGroup = new Model3DGroup();

            foreach (var scene in gltfModel.LogicalScenes)
            {
                var nodesToProcess = new Stack<(SharpGLTF.Schema2.Node node, Matrix3D transform)>();
                foreach (var node in scene.VisualChildren)
                {
                    nodesToProcess.Push((node, Matrix3D.Identity));
                }

                while (nodesToProcess.Count > 0)
                {
                    var (node, parentTransform) = nodesToProcess.Pop();
                    var transform = parentTransform * GetNodeTransform(node);

                    if (node.Mesh != null)
                    {
                        var mesh = node.Mesh;

                        foreach (var primitive in mesh.Primitives)
                        {
                            var positions = primitive.GetVertexAccessor("POSITION").AsVector3Array();
                            var normals = primitive.GetVertexAccessor("NORMAL")?.AsVector3Array();
                            var texCoords = primitive.GetVertexAccessor("TEXCOORD_0")?.AsVector2Array();

                            var mesh3D = new MeshGeometry3D
                            {
                                Positions = new Point3DCollection(positions.Select(v => new Point3D(v.X, v.Y, v.Z))),
                                TriangleIndices = new Int32Collection(primitive.GetIndices().Select(i => (int)i))
                            };

                            if (normals != null)
                            {
                                mesh3D.Normals = new Vector3DCollection(normals.Select(n => new Vector3D(n.X, n.Y, n.Z)));
                            }

                            if (texCoords != null)
                            {
                                mesh3D.TextureCoordinates = new PointCollection(texCoords.Select(t => new Point(t.X, t.Y)));
                            }

                            Color baseColor = Colors.Red;
                            Color emissiveColor = Colors.Black;
                            DiffuseMaterial diffuseMaterial = null;

                            if (primitive.Material != null)
                            {
                                var baseColorChannel = primitive.Material.FindChannel("BaseColor");
                                if (baseColorChannel.HasValue)
                                {
                                    var baseColorFactor = baseColorChannel.Value.Parameter;
                                    baseColor = Color.FromArgb(
                                        (byte)(baseColorFactor.W * 255),
                                        (byte)(baseColorFactor.X * 255),
                                        (byte)(baseColorFactor.Y * 255),
                                        (byte)(baseColorFactor.Z * 255));
                                }

                                var emissiveChannel = primitive.Material.FindChannel("Emissive");
                                if (emissiveChannel.HasValue)
                                {
                                    var emissiveFactor = emissiveChannel.Value.Parameter;
                                    emissiveColor = Color.FromArgb(
                                        255,
                                        (byte)(emissiveFactor.X * 255),
                                        (byte)(emissiveFactor.Y * 255),
                                        (byte)(emissiveFactor.Z * 255));
                                }

                                var textureChannel = primitive.Material.FindChannel("BaseColor");
                                if (textureChannel.HasValue && textureChannel.Value.Texture != null)
                                {
                                    var texture = textureChannel.Value.Texture.PrimaryImage;
                                    var bitmapImage = LoadImageFromGltf(texture);
                                    diffuseMaterial = new DiffuseMaterial(new ImageBrush(bitmapImage));
                                }

                            }

                            var baseMaterial = new DiffuseMaterial(new SolidColorBrush(baseColor));
                            var emissiveMaterial = new EmissiveMaterial(new SolidColorBrush(emissiveColor));
                            var combinedMaterial = new MaterialGroup();
                            combinedMaterial.Children.Add(diffuseMaterial ?? baseMaterial);
                            combinedMaterial.Children.Add(emissiveMaterial);

                            var geometryModel = new GeometryModel3D(mesh3D, combinedMaterial);
                            geometryModel.Transform = new MatrixTransform3D(transform);
                            _modelGroup.Children.Add(geometryModel);
                        }
                    }

                    foreach (var childNode in node.VisualChildren)
                    {
                        nodesToProcess.Push((childNode, transform));
                    }
                }
            }

            if (_modelVisual != null)
            {
                helixViewport.Children.Remove(_modelVisual);
            }

            _modelVisual = new ModelVisual3D { Content = _modelGroup };
            helixViewport.Children.Add(_modelVisual);

        }
        catch (Exception ex)
        {
            MessageBox.Show($"An error occurred while loading the GLB model: {ex.Message}");
        }
    }

    private BitmapImage LoadImageFromGltf(SharpGLTF.Schema2.Image image)
    {
        using (var memoryStream = new MemoryStream(image.Content.Content.ToArray()))
        {
            var bitmapImage = new BitmapImage();
            bitmapImage.BeginInit();
            bitmapImage.CacheOption = BitmapCacheOption.OnLoad;
            bitmapImage.StreamSource = memoryStream;
            bitmapImage.EndInit();
            return bitmapImage;
        }
    }

    private Matrix3D GetNodeTransform(SharpGLTF.Schema2.Node node)
    {
        var matrix = Matrix3D.Identity;

        if (node.LocalMatrix != null)
        {
            var mat = node.LocalMatrix;
            matrix = new Matrix3D(mat.M11, mat.M12, mat.M13, mat.M14,
                                  mat.M21, mat.M22, mat.M23, mat.M24,
                                  mat.M31, mat.M32, mat.M33, mat.M34,
                                  mat.M41, mat.M42, mat.M43, mat.M44);
        }

        return matrix;
    }
}

}

this is my entire logic to render .glb model with color and textures and it is working absolutely fine but now i wanted to support animated .glb files also what should i do to further complete this additional development of rendering animated .glb models

[vpenades]

Well... animation is all about replacing the "static" transforms by animated transforms, which is usually done by replacing the LocalTransform property for GetLocalTransform(animationName, time);

But this is how it is done at low level

The recomendation is to use the APIs provided by SharpGLTF.Runtime library, which is a helper class that can be used to render models and automates some aspects like animation, but more importantly, it handles some non obvious hidden details of animation.

[amit11996]

can you give a sample code which can give me an overview of how to render animated .glb models i mean i've seen that animation.channels has all kinds of value in key value pair for each node which can be used to perform animation but not understanding further how to use this data to render the animated glb model please provide me some guidance.