Animation Playback

[kklouzal]

I'm not sure what I am missing, attempting to use the 'Animation Playback' example ends up leaving my model in it's T-Pose and just kind of having a seizure.

    void updateUniformBuffer(const uint32_t &currentImage) {
        endFrame = std::chrono::high_resolution_clock::now();
        deltaFrame = std::chrono::duration<double, std::milli>(endFrame - startFrame).count();
        startFrame = endFrame;

        ubo.model = Model;
        ubo.Animated = true;

        controller_.Update(animation_, deltaFrame);

        //  Samples optimized animation at t = animation_time_
        ozz::animation::SamplingJob sampling_job;
        sampling_job.animation = &animation_;
        sampling_job.context = &context_;
        sampling_job.ratio = controller_.time_ratio();
        sampling_job.output = make_span(locals_);
        if (!sampling_job.Run()) {
            printf("Sampling Job Failed\n");
            return;
        }

        //  Converts from local space to model space matrices
        ozz::animation::LocalToModelJob ltm_job;
        ltm_job.skeleton = &skeleton_;
        ltm_job.input = make_span(locals_);
        ltm_job.output = make_span(models_);
        if (!ltm_job.Run()) {
            printf("LocalToModel Job Failed\n");
            return;
        }

        auto joints = skeleton_.num_joints();
        //printf("UBO Joints %i\n", joints);
        for (int i = 0; i < joints; i++) {

            ubo.bones[i] = models_[i] * InverseBindMatrices_[i];

        }
        //  Send updated bone matrices to GPU
        _Mesh->updateUniformBuffer(currentImage, ubo);
    }

I am obviously not doing something right when translating the OZZ LocalToModelJob over into GLM Matricies. Can you point me in the correct direction? My shader is very simple to animate.

#version 450

layout(location = 0) in vec3 inPosition;
layout(location = 1) in vec3 inColor;
layout(location = 2) in vec2 inTexCoord;
layout(location = 3) in vec3 inNormal;
layout(location = 4) in vec4 inBones;
layout(location = 5) in vec4 inWeights;
layout(location = 6) in vec3 inTangent;

layout(location = 0) out vec3 outNormal;
layout(location = 1) out vec2 outUV;
layout(location = 2) out vec3 outColor;
layout(location = 3) out vec4 outWorldPos;
layout(location = 4) out vec3 outTangent;

layout(std140, push_constant) uniform CameraPushConstant {
    mat4 view_proj;
    vec4 pos;
} PushConstants;

layout(std140, binding = 0) uniform UniformBufferObject {
    mat4 model;
    mat4 bones[64];
    bool animated;
} ubo;

void main() {
    //outWorldPos = ubo.model * vec4(inPosition, 1.0);

    if (ubo.animated)
    {
        mat4 skinMat =
            inWeights.x * ubo.bones[int(inBones.x)] +
            inWeights.y * ubo.bones[int(inBones.y)] +
            inWeights.z * ubo.bones[int(inBones.z)] +
            inWeights.w * ubo.bones[int(inBones.w)];

        vec4 Pos = skinMat * vec4(inPosition, 1.0);

        gl_Position = PushConstants.view_proj * ubo.model * Pos;
    } else {
        gl_Position = PushConstants.view_proj * ubo.model * vec4(inPosition, 1.0);
    }

    outUV = inTexCoord;

    mat3 mNormal = transpose(inverse(mat3(ubo.model)));
    outNormal = mNormal * normalize(inNormal);
    outTangent = mNormal * normalize(inTangent);

    outColor = inColor;
}

[guillaumeblanc]

Hi,

I can't point out something that explains the Tpose, but I can share a few points:

• I don't see where/if you multiply joint by inverse bind pose matrices. This is required for skinning, as you can see in ozz examples: https://github.com/guillaumeblanc/ozz-animation/blob/master/samples/skinning/sample_skinning.cc#L107
• Your shader uses inWeights.y 3 times, instead of z and w. Is that intended ? It doesn't explain the tpose, but can definitely have an impact.
• You shouldn't use m128_f32 member directly, that's not portable. You should use store and load functions, like in this example : https://github.com/guillaumeblanc/ozz-animation/blob/7fd3642f170f8ecaa31e911267e838eaf149b462/samples/framework/internal/shader.cc#L235

Hope it helps, Guillaume

[kklouzal]

Yeah that actually helps a lot, thank you!

In the sample_skinning.cc file it says // Mesh archive can be specified as an option.

can I export a mesh.ozz using the gltf2ozz tool? Don't see a switch for it in the --help output.

[kklouzal]

I have been able to simplify things by using native ozz::math::float4x4 instead of glm:mat4. I have also added the ability to pull the InverseBindMatrix for each joing out when loading my gltf files. Unfortunately, multiplying each joint matrix with its corresponding InverseBindMatrix results in the exact same result. I have updated my original post to include the changed code.

https://i.imgur.com/l4hb9v9.mp4

Do you have any idea where I should look to begin determining where I may be going wrong?

I really appreciate it!

[skaarj1989]

@kklouzal Have you tried to use the model (the one on video) in ozz framework (playback sample)?

With incorrect matrix multiplication your character would end up like a twisted creature from a Dead Space game:

---

layout(location = 4) in vec4 inBones;

Joints/bones are just integers. Why don't you use ivec4?

---

BTW I also use glm, here's a snippet that you might find useful:

glm::mat4 to_mat4(const ozz::math::Float4x4 &m) {
  glm::mat4 result{};
  for (uint8_t i{0}; i < 4; ++i)
    ozz::math::StorePtr(m.cols[i], &result[i].x);
  return result;
}

[guillaumeblanc]

gltf2ozz doesn't support meshes. It could, but it's not implemented.

Seems you're going to need to find strategies to debug indeed. Looking at your video, it looks like the mesh is skinned by a single joint (maybe??). You could try to:

• display joint index as a vertex color, just to see if multiple joints are affecting the skin.
• display weights as vertex colors, again to check data make sense.
• draw joints' position
• ...

One usual mistake/complexity is to have a mismatch between joint indices/order (skeleton/animation vs inverse bind poses vs mesh). This is especially easy to break if you really on multiple importers. It's a bit complex to debug, but as long as your mesh is static/t-pose, I think you have something else to fix first.

Guillaume

[kklouzal]

Thank you guys, yes I indeed need some debugging tactics. I started rechecking what I had again and found I wasn't properly loading the bone data. Now I've got something that looks similar to what @skaarj1989 showed https://i.imgur.com/FRsSbYx.mp4 odd he moves properly, but the vertices just seem to be incorrectly mapped on some places of his body. I tried using the animation.ozz and skeleton.ozz inside the sample playback project and the skeleton looks/moves properly. Any ideas here? It's obviously the matrices, possibly out of order? But then why would he be moving properly?

[guillaumeblanc]

Hi,

any progress since then ?

Cheers, Guillaume

[kklouzal]

No, i've done many things to try and solve this. I think some of the vertices may be out of order..? I'm not sure.. Took a couple weeks break to get my mind off of it and will go try again.. Thank you for asking though :)

I was really hoping someone with more experience than me would see my video and say "oh yeah that looks like XXX is happening"..

[jankrassnigg]

Unfortunately animations typically look all kinds of broken until you have everything 100% right. Just a single tiny mistake usually results in something completely bonkers. That's why it's really hard to look at these bugs and tell you what you've missed. Also there are just soo many pitfalls.

Since your animation is already playing pretty well. I do think your code is mostly correct. You should try simpler meshes to debug this issue. For instance, built a simple model (e.g. two cubes) in Blender with just two bones and a trivial animation around one axis. If you don't yet have the skills to model something like that, my #1 tip would be to learn that first. Takes a couple of days, but is absolutely worth it. It is possible that your character uses more than 4 bones per vertex for skinning and your code only uses up to 4. In that case single vertices may lag behind, because they are missing a percentage of their transformation. To fix that you can find the 4 strongest bones weights and re-normalize them. However, I would start with just asserting that your character does not have such data in the first place, and use a different model, if it does, because it's possible that you run into other problems with that model as well. In my experience some models stop working properly when you normalize the bone weights. So in general, make sure that you have a test model that satisfies your assumptions first, then debug your code. I've wasted a lot of my time because I had only few meshes for testing and they had various idiosyncrasies that needed different work-arounds.

Another thing: Make sure the bone data only contains rotations (to begin with). Translations are usually fine as well, though rarely needed. Scaling of bones introduces all sorts of issues, because if used by an artist, it's often non-uniform scaling. For RENDERING that actually may work, but once your want to do more (e.g. physics ragdolls) it becomes an absolute nightmare.

You should also check that all bone indices are within valid ranges. If a GPU tries to read a bone outside the valid range, it usually "just works" and doesn't crash, so it's not as easy to realize. In theory all bone weights at one vertex should sum up to 1. In practice that's not the case for all models. Either scan your data first to validate that this is the case, or build your own test mesh, where you can be certain about this.

You should also write a shader to visualize the available data:

• Write the 4 bone weights to the color output. If single vertices have broken data, it will become apparent.
• Write the 4 bone indices to the color output (with some remapping into 0-1 range). Again, if a vertex has corrupt data, it will be apparent.

Also make sure that during data upload to the GPU you are sending ALL the data. It's easy to accidentally not copy all the bones or weights, due to off-by-N errors. E.g. if you have 13 bones, you may only be uploading 12, because that's the closest number divisible by 4.

Those are some of the pitfalls from the top of my head. Good luck! Also, if you need some more inspiration ezEngine is using ozz behind the scenes. It's a lot of complex code, so not sure whether it's really helpful. But you could try to load your model there and see what happens. Btw. double checking with other apps (e.g. the Windows model viewer or Unity) is always a good idea to verify that your input data is not too much screwed up.

Cheers!

[kklouzal]

I implemented @skaarj1989 ozz->glm matrix conversion function to make sure nothing was getting messed up there. I pulled out the animation matrices from the objects uniform buffer and stuck it in its own storage space buffer to make sure the data wasn't getting cut off/mangled there. Visualized the joint weights and joint indices via shader code and compared it to an example application that uses the same model, everything matches there.. (modified my shader and the example shader to colorize the model the same way). Verified in blender the model uses 4 bones per vertex and the model works in the example application (sadly the example does not use OZZ) https://github.com/SaschaWillems/Vulkan/tree/master/examples/gltfskinning I modified my gltf file loading code to match that used in the saschawillems example. No change after modification aside from a speed increase due to my naive/lazy implementation lol.. Also, the model displays properly until I turn animations on..

So, I think I can safely deduce that I'm loading the model properly into my renderer. I feel like the disconnect is now within my implementation of OZZ, perhapse due to the fact that I am using the gltfTOozz tool provided by the library? I think @guillaumeblanc has mentioned that tool is slightly lacking compared to the fbxTOozz tool?

Question, my application simply takes OZZ LocalToModelJob output and multiplies each output joint by the joints inverse bind matrix: to_mat4(models_[i]) * InverseBindMatrices_[i] In the many skinning examples I've looked through, they also make use of an 'inverse transform'? Does OZZ take care of this piece inside the LocalToModelsJob correct?

I don't know what else is wrong aside from the joint transformations I'm passing over being incorrect? OR The joint indices are mixed up from OZZ compared to the order in which is loaded directly from the GLTF file?

I am thinking I will code into my application directly the functions gltfTOozz provides to ensure the last point.. I was going to do this anyways..but wanted to ensure the model animated properly before adding another layer of complexity..

My inclination is that gltfTOozz tool output is mangling something, dunno..

Thank you again everyone for your input thus far.

[kklouzal]

Now that I'm looking more closely at my model as it animates, it does indeed look like the order of bones is different between what I use from the GLTF file and what the LocalToModelsJob outputs.. hence why it's moving properly but looks like it went through a meat grinder. Some of the bones match making it look halfway correct in some places. Looking at it in more detail, for example, the left foot is in the right foots position and the triangles get stretched as a result, etc..

[kklouzal]

Yeahh.... A couple of the joints are out of order, so simple iterating sequentially through the array of joints will end up using incorrect transformations for some bones... Well I'm pretty sure fixing this ordering will solve my problems.. Creating a map of BoneID->Bone_Name should suffice, I'm trying to figure out how to get the name out of the gltf file using tinygltf, once I pin that down it should be trivial to get this sorting figured out...

[jankrassnigg]

Somewhere in the pipeline (maybe not your code, maybe in the gltf import) it probably uses the SkeletonBuilder class (I use my own code for data import, so not sure what the ozz tools do). That thing will rearrange bones such that they can be updated in a linear fashion, meaning when it updates bone N, it can read all bones N-x, because they are guranteed to already be up to date.

That may be why you see a different order. It is correct though. It is possible that you copy the bone index straight from the GLTF data and are missing a remapping step. I really can't remember much of this, but I guess the skeleton class must have a method to tell you what the rearranged index is.

Hope that helps.

[kklouzal]

So I decided to manually map the offsets by hand. And look it animates properly!

        //
        //  Manually map the sequential order of bones from GLTF
        //  over to the jumbled up sorting we get from OZZ.
        std::vector<int> JointCorrections;
        JointCorrections.push_back(0);
        JointCorrections.push_back(1);
        JointCorrections.push_back(2);
        JointCorrections.push_back(3);
        JointCorrections.push_back(4);
        JointCorrections.push_back(5);
        JointCorrections.push_back(8);
        JointCorrections.push_back(6);
        JointCorrections.push_back(9);
        JointCorrections.push_back(7);
        JointCorrections.push_back(10);
        JointCorrections.push_back(11);
        JointCorrections.push_back(15);
        JointCorrections.push_back(12);
        JointCorrections.push_back(16);
        JointCorrections.push_back(13);
        JointCorrections.push_back(17);
        JointCorrections.push_back(14);
        JointCorrections.push_back(18);

        //
        //  Iterate through OZZ LocalToModel job
        //  and build a vector of joint matrices
        auto joints = skeleton_.num_joints();
        std::vector<glm::mat4> Jnts;
        for (int i = 0; i < joints; i++)
        {
            //  Store the transformation matrix of each joint (correcting OZZ array position)..
            Jnts.push_back(to_mat4(models_[JointCorrections[i]]) * InverseBindMatrices_[i]);
        }

So.. now I'm left trying to figure out how to get the char/string name of each joint directly out of the GLTF file OR if OZZ has a way to 'unscramble' it..

[jankrassnigg]

Ozz does have all the necessary information for you, you just need to find it. Sorry can't help more, but @guillaumeblanc should know those details.

[guillaumeblanc]

Hi,

That's great news !!!

ozz skeleton builders orders joints depth first, but it's an internal detail. You should be able to implement your solution without relying on that. So it makes sense that you have to match joints by name, then either reorder skinning matrices or fix up joints indices while loading the mesh.

ozz fbx mesh importer is doing it also: https://github.com/guillaumeblanc/ozz-animation/blob/master/samples/framework/tools/fbx2mesh.cc#L432. That's why the mesh importer takes the skeleton as input: https://github.com/guillaumeblanc/ozz-animation/blob/master/samples/framework/tools/fbx2mesh.cc#L46. I think that's the cleanest solution.

Hope it helps, Guillaume

[kklouzal]

Yeah that was the approach I wanted to take here. I'm probably going to have to build a map on my GLTF loaders side that maps a vector of integers, in a depth first order, starting at the root node. That way I will hopefully have indices that match OZZ. Unfortunately, GLTF files do not always store the node/joint names, so keeping a map of names->indices won't always work...

Would it be possible to add a map/span to the ozz::animation::Skeleton maybe <std::string, int>' or simply an 'ozz::span<uint16_t>?

Could be like

ozz::animation::Skeleton MySkeleton;
MySkeleton.joint_indicies;

This way we can iterate that variable, and outputs the corresponding joint index. It would eliminate the need to keep a map of joint names on hand for this very problem.

        //
        //  Iterate through OZZ LocalToModel job
        //  and build a vector of joint matrices
        auto joints = skeleton_.num_joints();
        std::vector<glm::mat4> Jnts;
        for (int i = 0; i < joints; i++)
        {
            //  Store the transformation matrix of each joint (correcting for OZZ depth first array positioning)..
            Jnts.push_back(to_mat4(models_[i]) * InverseBindMatrices_[skeleton_.joint_indicies[i]]);
        }
        //
        //  Send updated uniform buffer to GPU
        _Mesh->updateUniformBuffer(currentImage, ubo);
        //
        //  Send updated bone matrices to GPU
        if (Jnts.size() > 0)
        {
            _Mesh->updateSSBuffer(currentImage, Jnts.data(), Jnts.size() * sizeof(glm::mat4));
        }

Then when loading the GLTF file, I just have to store the InverseBindMatrices in a top-down order, or keep a map on hand of InverseBindMatrix_Position->Joint Index (or vise versa).

What do you think @guillaumeblanc ?

[kklouzal]

Yeah I'm afraid there is no way to get the node name paired with the joint index for a GLFW file. You have to traverse the nodes to grab their names but you have to traverse the skin to get the joints. The ID's between these two differ and don't align.

Model->Node->Name Skin->Joint->InverseMatrix

The indices do not align.

OZZ needs a way to store the original joint indices in the skeleton, simply having the name does not suffice. Maybe for FBX but not GLFW..

[kklouzal]

@jankrassnigg Does your ezEngine use gltf files with ozz? I'm struggling to find any way to map the joint index to the joint name. On the OZZ side this is pretty straight forward, on the GLTF side not so much.. Seems like the joint names are stored inside the node and the joint indices are inside the skin. Unfortunately it seems like the Nodes are stored in a different order than the Joints so things still don't align...

[jankrassnigg]

We use https://github.com/assimp/assimp for the asset import, which does support gltf, yes. I don't remember this part to have been an issue. But then there were lots of problems to figure out, so not sure.

[kklouzal]

Put this issue on hold, for now, and went to work on other aspects of my engine. I will return to this problem shortly. Nonetheless haven't been able to solve it yet. OZZ scrambling the index ordering is definitely causing headaches on this. An internal OZZ structure with the original index ordering would be helpful.

[guillaumeblanc]

An internal OZZ structure with the original index ordering would be helpful.

I can't make sense of "original index ordering" concept. Ozz skeleton format is independent of the file/format from which it was imported.

Seems like the joint names are stored inside the node and the joint indices are inside the skin.

In this case it seems possible to map gltf nodes indices to ozz joints indices, matching by name, and later replace joint indices in the skin to ozz indices. Note that inverse bind poses need the same remapping. I strongly suggest to look at that part of the implementation in fbx importer (https://github.com/guillaumeblanc/ozz-animation/blob/master/samples/framework/tools/fbx2mesh.cc#L432). I'd do the same strategy if I were to implement a gltf mesh importer, or any mesh importer actually.

Hope it helps, Guillaume

[kklouzal]

Mapping gltf joint names to joint index was easier once examining the specification. https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.pdf Section 3.7.3.2. Joint Hierarchy:

The joint hierarchy used for controlling skinned mesh pose is simply the node hierarchy, with each node designated as a joint by a reference from the skin.joints array Which, from what I understand, means a skins joint id == node id.

for (size_t i = 0; i < model.skins.size(); i++)
{
tinygltf::Skin _Skin = model.skins[i];

for (auto& JointID : _Skin.joints)
{
    std::string NodeName = model.nodes[JointID].name;
    int ID = JointID - _Skin.skeleton;
}

}

Subtract `_Skin.skeleton` from each `JointID` in the event our skeleton root node is greater than 0. If you don't do this then your joint ID's will be skewed and won't match up with the inverse bind matrices.

Now I have an array of inverse bind poses and an array of joint names, both match up to the correct joint index. Still getting a mangled monstrosity when trying to display the 3d model.

auto joints = skeleton_.num_joints(); for (int i = 0; i < joints; i++) { const char* OZZJointName = skeleton.joint_names()[i]; uint16_t GLFW_JointIndex = _GLTFJointMap[OZZ_JointName];

glm::mat4 OZZ_Matrix = to_mat4(models_[i]);
glm::mat4 GLFW_Matrix = InverseBindMatrices_[GLFW_JointIndex];

if (_Mesh->bAnimated && i < 32)
{
    _Mesh->instanceData_Animation[instanceIndex].bones[GLFW_JointIndex] = OZZ_Matrix * GLFW_Matrix;
}

}

![image](https://user-images.githubusercontent.com/2928845/193714614-1955f51c-00f7-49ba-beba-4eb3d6fa4cee.png)

This is extremely frustrating. I know I'm just not doing something right. Which is the most frustrating part.

[guillaumeblanc]

Yes it's frustrating! Skinning has so many parameters, it's difficult to debug.

Now I have an array of inverse bind poses and an array of joint names, both match up to the correct joint index.

How about joint indices in the skinned mesh? They shall be "reordered" also to match with ozz skeleton joints order too. Have you done that ?

You could also reorder matrices in instanceData_Animation[instanceIndex].bones, both strategy work. The important thing is that the order of the matrix array is coherent with the order of joints in the skin.

Hope it helps, Guillaume

[guillaumeblanc]

Any progress ?