Importing a skinned mesh from blender .fbx exported file

[victorcifuentes]

I'm trying to import a skinned mesh from a .fbx file exported from blender, using fbx2mesh.cc framework sample, with a custom raw skeletal, but it results in something as follow:

The original mesh in blender looks like:

I do not have much experience, but a think this happens by bones vs joints oriented format issue. And I don't know how to solve this.

Thanks in advance.

[guillaumeblanc]

Hi,

it's a bit difficult to debug with a screenshot, but I can try to guide you.

First, could you explain what you mean by:

• bones vs joints oriented format issue ?
• with a custom raw skeletal ? I presume you don't import the skeleton, but build it by code. Do you still pass your skeleton to fbx2mesh command line tool? It's required for the importer to correctly index joints.

Then I'd recommend to check a few things (if you didn't already):

• Display the skeleton and check it's correct.
• Display the mesh unskinned and check it's correct.
• Skin the mesh with identity skinning and inverse bind-pose matrices.

If we know all of this works, we can get closer to the issue.

Hope it helps, Guillaume

[victorcifuentes]

Hi! I sorry, I'll try to explain myself better.

With "Bones vs joints" oriented format, I means that when I build the armature using Blender, I named bone by bone, because Blender is oriented to bones. Instead, when I write the code, I name joint by joint. I suspect maybe this is the problem, In other words, in Blender the the root is a bone and in Ozz it's a joint.

I don't use fbx2mesh command line tool, instead, I build my own code using fbx2mesh.cc code.

Displaying only the skeleton works ok, even with a custom animation.

I don't know how to display the mesh unskinned, but I open the mesh in Unity from .fbx file and it's right.

[guillaumeblanc]

Hi,

• "Bones vs joints": I don't think it's your issue, because in my opinion it's just a naming convention.

• "I don't use fbx2mesh command line tool": I presume you're still passing your skeleton to fbx2mesh functions. BuildSkin function will detect naming issues and fail from there. If it passes that's a good step already.

• "I don't know how to display the mesh unskinned": If you're using ozz sample framework, then simply call DrawMesh, instead of DrawSkinnedMesh. Maybe then you can display the skeleton on top of the mesh to check they fit one with each other.

Overall I think you need to check step by step that everything is fine, hence my suggestions. You could also simplify your data to a single face and two bones (maybe even one), it'll be easier to debug and understand. Another idea is to import the skeleton from the fbx (using fbx2ozz). It might not be your goal, but still a debug step to understand your issue.

Hope it helps, Guillaume

[victorcifuentes]

Hi! Thanks for your help.

Reading the web, I could solve the issue shown in my first screenshot. I just needed to change parameter's into blender fbx export module (Scalings: "FBX Units scale", Forward: "-Z", Up: "Y").

Now I can show the mesh and the posture separately skipping skinning.

But, when I try to skinning, it seems like all joints references were wrong. As show in the follow screenshot, thigh mesh start skinning at knees, and legs start at ankle.

My blender pose armature is as follow, and in the same way I writed the skeletal structure on Ozz, but by joints:

[guillaumeblanc]

Hi,

Any progress on your side ?

From what I can foresee:

• a joint indexing issue (joint order in the skeleton doesn't match with joint indexes in the mesh). If you provide your skeleton to fbx2mesh function and it doesn't return a failure it should be fine.
• wrong inverse bind poses. Bind poses are extracted by fbx2mesh so it shouldn't be an issue. The only thing I can think is if your skeleton doesn't conform with the one used to setup the mesh in blender.

Which leads me to something strange in your screenshot. It seems that skeleton joints are not rotated (the 3d gizmo on each joint seems axis aligned), but traslated. Are you translating joints in your animation or skeleton? If this is the case I'm not sure skinning is going to work fine. It's most probably no conforming with bind pose transformation coming from blender.

I don't know why your generating the skeleton from code, but you might consider importing it (fbx2ozz) just as a matter of testing, temporarily.

Cheers, Guillaume

[victorcifuentes]

Hi!

I finally did it that you suggest me. I imported both skeleton and mesh using the command line tools, and yes! It's works well. :)

As you say, I'm making a custom animation based in translating joints and surely it's one of fail. I'm currently trying to repoduce information captured from inertial measure units (IMUs)

On the other hand, I need to build the skeleton from code, because i'm reading real anthropometric data and the proporsions should be preserved.

But you're right, maybe the issue is with the joints's structure, but I do not found why.

Now, I go to work in the joints's rotations, and the initial bind pose to trying to contruct a right animation.

Thanks again !

[guillaumeblanc]

Hi,

Good job! Importing the skeleton might not be your goal, but at least it shows what's working and what's not.

Building your skeleton should work as well. Maybe you could take advantage of the imported one to see how it's built. Using fbx2ozz you can output the raw skeleton, which will be easier to read/understand.

To help you with switching from using translations to rotations, you might find useful to use this functions that build a quaternion from two vectors. Note it's still on feature/ik branch only.

I'm curious, what's the output of an IMU: a force, an angular speed... ?

Cheers, Guillaume

[victorcifuentes]

Hi,

The IMU data depends of what kind of IMU are you using, it can be both acceleration and angular speed for each axis and even magnetic fields. I'm just using angular speed to my purposes.

I'm still working on custom animation, without success. When I import both mesh and bind pose from fbx file it works OK.

But, if I blend It with a sampling job which contains:

1) An empty animation (without keys) the mesh appears caked on a single point around of (0,0,0) in other words bind pose is null or something else.

2) An animation with only one Translation Key for each joint (trying to recreate bind pose) mysteriously it looks like my first screenshot (like spaghetti). But skeletal bind pose is OK.

3) An animation with all data using Translations Keys, the animation runs OK, but the skinned mesh is still like a spaghetti.

My OnUpdate function looks like:

controller_.Update(*animation_, _dt);
ozz::animation::SamplingJob sampling_job;
sampling_job.animation = animation_;
sampling_job.cache = cache_;
sampling_job.ratio = controller_.time_ratio();
sampling_job.output = locals_;

if (!sampling_job.Run()) {
  return false;
}

// Prepares standard blending layers.
ozz::animation::BlendingJob::Layer layers[1];
layers[0].transform = locals_;
layers[0].weight = 1.f;

// Setups blending job.
ozz::animation::BlendingJob blend_job;
blend_job.layers = layers;
blend_job.bind_pose = skeleton_.bind_pose();
blend_job.output = blended_locals_;

// Blends.
if (!blend_job.Run()) {
  return false;
}

// Converts from local space to model space matrices.
ozz::animation::LocalToModelJob ltm_job;
ltm_job.skeleton = &skeleton_;
ltm_job.input = blended_locals_;
ltm_job.output = models_;

if (!ltm_job.Run()) {
  return false;
}

And my OnDisplay

`assert(models_.count() == skinning_matrices_.count() &&
       models_.count() == mesh_.inverse_bind_poses.size());

// Builds skinning matrices, based on the output of the animation stage.
for (size_t i = 0; i < models_.count(); ++i) {
  skinning_matrices_[i] = models_[i] * mesh_.inverse_bind_poses[i];
}

_renderer->DrawPosture(skeleton_, models_,
                                ozz::math::Float4x4::identity());

// Renders skin.
return _renderer->DrawSkinnedMesh(mesh_, skinning_matrices_,
                                  ozz::math::Float4x4::identity(),
                                  render_options_);`

The first translations that I use is something like:

raw_animation.tracks[0].translations.push_back({ .0f, ozz::math::Float3(0.f, .25f + (dim_thigh+dim_leg) * factor, 0.f)}); raw_animation.tracks[1].translations.push_back({ .0f, ozz::math::Float3((dim_huckle / 2.0f) * factor, -0.15f, 0.05f)}); raw_animation.tracks[2].translations.push_back({ .0f, ozz::math::Float3(-(dim_huckle / 2.0f) * factor, -0.15f, 0.05f)}); raw_animation.tracks[3].translations.push_back({ .0f, ozz::math::Float3(0.f, -(dim_thigh * factor), 0.f)}); raw_animation.tracks[4].translations.push_back({ .0f, ozz::math::Float3(0.f, -(dim_leg * factor), 0.f)}); raw_animation.tracks[5].translations.push_back({ .0f, ozz::math::Float3(0.f, -0.1f, (dim_foot * factor))}); raw_animation.tracks[6].translations.push_back({ .0f, ozz::math::Float3(0.f, -(dim_thigh * factor), 0.f)}); raw_animation.tracks[7].translations.push_back({ .0f, ozz::math::Float3(0.f, -(dim_leg * factor), 0.f)}); raw_animation.tracks[8].translations.push_back({ .0f, ozz::math::Float3(0.f, -0.1f, (dim_foot * factor))});

Please, tell me if you find something wrong, or you need that I show you something else.

Thanks !

[guillaumeblanc]

Hi,

I think you can't animate a human skeleton with translations. I see translations as the way to give bones a length. This length is fixed (constant all along animation): bones don't change of length when walking. Only joint rotation change while walking. It's coherent with skinning: You don't want mesh vertices to translate, you want them to rotate around joints.

An animation should never be empty. It should at least contain 1 translation key frame per joint to fix bone length. A bone is usually along a single axis, so it means a rotation key is needed per joint to put the bone in the right direction. I don't know how your skeleton/mesh was setup, but that's where you should look for an answer there. If the mesh was skinned with rotated joints but the animation isn't, you definitely ends with spaghetti as you say.

I can recommend 2 ideas:

1. Find in blender what are the rotation and translation of each joint/bone of you skeleton, and copy paste them when building your skeleton and animation by code.
2. Or when you import the skeleton from fbx, extract those information from the bind pose, which contains this information. It's a bit more tricky because you have to understand bind pose content which is a SoA data structure, but it's doable. On the other hand you know these data are correct (scale, coordinate system...) because you validated them with your tests already.

Either way, you will have the data to build the skeleton and a static/fix animation. Then you'll "only" have to animate joint rotations, multiplying bind pose ones with the rotation you get from IMU.

Hope it helps.

Cheers, Guillaume

PS: Displaying the skeleton looks ok but it doesn't mean it's correct. Bones are rendered in between 2 joints, so only model space translations are used. But the rest of the matrix is important for skinning.

PPS: You don't need to blend if you have a single layer. The output of the blending job will be a copy of the input.

[guillaumeblanc]

Just remember that using fbx2ozz you can output the raw skeleton and animation. That means that if you import that in your code, you can inspect the default pose your skeleton should have. So it's kinda idea 2 without needing to worry about SoA.

I understand you need to build the skeleton / animation from code, but you need to get the default joint rotations that work with your skinned mesh. Also you'll need to animate rotations, just like human do.

[guillaumeblanc]

I hope you manage to do what you wanted. Don't hesitate to reopen this issue if you need more help on the topic.