Skinning + Morph Targets, only minor refactoring needed (sic!)

[apendua]

I finally managed to get it to work :) It turned out that the main obstacle in making these two types of animation cooperate with each other has always been the fact that the boneMatrices passed to the shader represent bones' transforms in global (world) space, while it seems more appropriate to pass the relative offsets of bones, i.e. bone's skinMatrix, with respect to the bone's rest position. If this approach is taken, the process of skinning vertex reduces to

vec4 skinned = boneMatX * skinReference * skinWeight.x;
skinned += boneMatY * skinReference * skinWeight.y;

where skinReference represents the "original" position of a vertex. So, not only does this approach allow us to use the morphed vertex as the skinReference in the skinning procedure (yes, the morph targets should go first), but also it denies any need of using the (large!) skinVerticesA/B arrays. This should be considered as a huge advantage, at least as long as someone needs to add more bone influences (still marked as TODO somewhere in the code).

A working example may be found here: (the model is pretty high poly, so please ignore any low frame rate issues)

http://oniostudio.com/skinning/

By the end of the week I should be able to clean up the code. The changes also include modified normal vectors (for basic shaders) and correct shadow maps generating. If you are interested in applying these changes to the project, please let me now what can I do to help in this process.

[mrdoob]

Uh, it's not working here...

Here's the log: http://pastebin.com/WRduaxvj

[apendua]

It should work now, sorry for that :) By the way, I've been testing it on chromium and still have some issues on FF.

[mrdoob]

Uhm... this is crashing webgl on the browser. Maybe this is using a version of the library that had the anisotropic on the shadowmap bug?

[apendua]

One more update... fixed... :)

Turning shadow maps off stops my FF from crushing all the time, so indeed it may be connected with the bug you mentioned. Is this issue already solved? Maybe I should get a newer version of the code...

[mrdoob]

Yeah, try updating to the latest code in dev. It works now by the way.

[mrdoob]

What do you think about the suggestion @alteredq?

[alteredq]

This is something that I was also wondering about when checking how others do skinning - we are the odd ones, what @apendua suggests seems to be "standard" way.

Maybe @empaempa could shed some light on this? I remember fuzzilly he did have some reason for doing it like that.

[apendua]

@alteredq When I first looked into the code, my first impression was that your approach is a consequence of the fact, that the animation system already stores and animates bone transforms relative to their parents, not to the bone's rest position, which is another - almost equivalent - way of doing the job. But from my point of view, this "relative animation" is even better because, you don't have to compute the bone's offset in "rest coordinates" for each frame, which can clearly save you some time.

[empaempa]

Not sure I'm 100% following the discussion, but the animation format was defined by @drojdjou and I know I changed the posing and shader code back and forth a couple of times before ending up where it is now - and I'm sorry to say that I don't remember the reasoning behind it ;)

If there's a faster, better way - by all means, go for it :)

Edit: The road we went might have to do with not having the rest position in the format, but I might be mistaken on that.

[alteredq]

Ok, so it seems it would be worthwhile to try to integrate this into the lib fully.

We'll probably lose buffaloes in the process (there is no source model available anymore) but as long as we have Blender pipeline working, we could just use some other models for the skinning example. At minimum these simple humans I used here have ok license.

For the process - the best would be if you could merge your code (including corresponding Blender exporter changes) with the latest dev branch of the lib (or at least have all your changes available as some branch in your GitHub three.js repo).

Also it would be good to have some working example, starting from blend file up to the running animation, to be able to test that the pipeline works.

[mrdoob]

We'll probably lose buffaloes in the process (there is no source model available anymore)

Can't really say I'm going to miss them ;P

Just so you know... I still have to sort out the loading API and the build script, but I'll also wait until you guys are happy with skeletal animation before wrapping up r50.

[alteredq]

Just so you know... I still have to sort out the loading API and the build script, but I'll also wait until you guys are happy with skeletal animation before wrapping up r50.

We also still need to finish this UV unflipping business :S

As for skeletal animation for r50 - what is now in dev is working and should be self-contained, without loose ends (exporter + playback + shaders + shadows) so it's possible to release it like that and it would be already huge improvement over r49.

The downside would be that people would need to re-export their models for r51 but there is quite a chance this will happen anyways (animation is huge topic, there is so much more to be done).

[mrdoob]

We also still need to finish this UV unflipping business :S

What else is left? (Sorry for the off-topic)

The downside would be that people would need to re-export their models for r51 but there is quite a chance this will happen anyways (animation is huge topic, there is so much more to be done).

Yep. But we're certain that this change would require re-exporting. And r50 already requires re-exporting. But we aren't certain that if this landed in r50 there will be something that would require re-exporting for r51. There may be, but we don't know. So by waiting a bit we reduce two certain re-exports into one.

[alteredq]

What else is left? (Sorry for the off-topic)

Every geometry that is not plane / cube / sphere / cylinder (there are 17 geometries currently in extras).

So by waiting a bit we reduce two certain re-exports into one.

Well, if we wait for these changes to be at least up to par with current state, it may delay r50 by a while (just going by how long these things usually take).

And r50 is already overdue, it hurts me to see people still on r49 for so long :S. Actually, I would probably even move this loading and build stuff you wanted to do for r51.

[apendua]

the best would be if you could merge your code (including corresponding Blender exporter changes) with the latest dev branch of the lib (or at least have all your changes available as some branch in your GitHub three.js repo)

@alteredq Sounds like fun :) I'll let you know when it's done.

Also it would be good to have some working example, starting from blend file up to the running animation, to be able to test that the pipeline works.

Is the "knight" example acceptable? I've some more attack moves to offer. Or, if you prefer, I can provide some standard walk / run cycle or jump animation of another humanoid.

The downside would be that people would need to re-export their models for r51 but there is quite a chance this will happen anyways (animation is huge topic, there is so much more to be done).

@alteredq, @mrdoob I'm pretty much sure, that the changes to the skinning routines that I've made so far are not interfering with the export pipeline at all. But as @alteredq suggested there are definitely some other animation issues that would require reexporting in the future. By the way, can you say something about what particular issues were you thinking?

Well, if we wait for these changes to be at least up to par with current state, it may delay r50 by a while (just going by how long these things usually take).

Totally agreed... actually I was hoping to have all the merging work done by now, but it end up as usual :( I don't want to delay you guys, and I would even feel more comfortable if this whole skinning stuff was postponed to r51.

[mrdoob]

Cool. Sounds good to me.

Btw, I can try asking on twitter-land to see if someone wants to donate skinning tests.

[alteredq]

Is the "knight" example acceptable? I've some more attack moves to offer. Or, if you prefer, I can provide some standard walk / run cycle or jump animation of another humanoid.

The ideal would be to have animation set like MD2 models have: idle, walk, run, jump, attack, taunt, something that could be used for more than just a playback of single animation, something with what we could test transitions and blending later on.

But for merging and testing anything should do. It's more about just making sure that what worked once on your machine will survive the merge and still keep working on other people's systems, including asset pipeline.

But as @alteredq suggested there are definitely some other animation issues that would require reexporting in the future. By the way, can you say something about what particular issues were you thinking?

Could be many things, for example:

• we still don't really have any way to deal with rigid bodies animation, I didn't even start to think how this could interact with skinning animation system
• multiple animations in a single model JSON
• I think you mentioned bone constrains for IK, motion graphs
• our animation format is quite verbose, animations are taking a lot of space, maybe we could make it smaller somehow?
• for skinning + morphing it feels wasteful to have morphs for full mesh when for example just face is using morphing

Btw, I can try asking on twitter-land to see if someone wants to donate skinning tests.

That could be useful once we have more stabilized code and pipeline.

[apendua]

@mrdoob Done... the skinning code is ready to go and the pull request is sent :)

[apendua]

You can download example models from here: http://oniostudio.com/skinning/static/obj/

[OniDaito]

Hi Chaps. I've been looking at this and we need to get something into three.js from a 3DSMax workflow. My thinking is to use an FBX Export script that will plant in the bones and their rotations with a given timestamp. Does this seem sensible? Does FBX hold the bone rotations per second/milisecond/whatever that your JSON file format needs?

[mrdoob]

Maybe 3DSMax->Collada->Blender->JSON?

[OniDaito]

Hahaha! That is likely to be a little error prone I suspect. Im prepared to write my own FBX exporter (I have played with Skinning, FBX and Cinder before). All I need to be sure of is that the timing code (i.e, at time X, bone Y should have rotation Z) exists within the FBX.

[OniDaito]

Well, as there was no harm in trying I thought I'd give it a go. Sadly, I can't even get the static mesh up and running. It could be our asset but basically, there are no faces reported and I think Collada has messed up the bones somewhere. Tricky!

[alteredq]

You could also try 3DSMax->Collada->ColladaLoader.

I think our ColladaLoader is more robust than Blender's Collada importer, I had some models which failed to work in Blender showing up ok in three.js.

Just for skinning ColladaLoader creates morphs instead of SkinnedMesh.

[OniDaito]

Thanks for the feedback. No joy with Collada - generally whenever I hear the C word I get the fear. I've never had it work for me. Currently, the static mesh extraction seems ok. Moving on to getting the bones out of the FBX now

[OniDaito]

I have noticed that in Rome, you guys exported from Unity. FBX is obviously well supported. I believe I've found the exporter you guys used: https://github.com/toji/building-the-game/

Is this correct? Im having a few issues with it - it seems to export binary data. Can I use that for the vertices in a model?

[mrdoob]

Nope. We used a exporter that @drojdjou did for the project. But I think he never open sourced it.

[OniDaito]

Ah well, never mind. Thanks for the feedback. Can I query a couple of things please? the skinWeights are the weights per vertex (in the same order), the skinIndices map the vertices to their bone. The Bone pos, scl, rot & rotq refer to the global position, global scale, global rotation as xyz euler and global rotation as a quaternion respectively? Or is this the break down of the transformation matrix that the bone asserts on its vertices? I've managed to extract the basics from the FBX (bone names, parents and such) and I think I have the weights but not sure on these other values.

[OniDaito]

Which algorithm are you using for the skinning? Are we using smooth skinning with a set two weights for the start and end bones?

[DoodleIncident]

This is great, @apendua! This is exactly the capability our project needs.

All of our files are in 3D Studio Max, though. Is there a format that would let me pass the animation details into blender, or should I just write a Max Script?

[apendua]

@DoodleIncident Nice to hear that this feature turns out to be useful :D

All of our files are in 3D Studio Max, though.

I wouldn't recommend using blender in your pipeline then. Blender's bone system is kinda awkward if you compare it to more "standard" solutions like FBX. As far as I know, blender does not have a decent FBX importer. You can write one on your own, but keep in mind that translating everything one way or another can be a little painful (at least it was painful for me). On the other hand, going FBX->three.js should be pretty straightforward (i.e. it should translate directly without any complex calculations). If I understand well, @OniDaito is already working on this one, so maybe you could team up, or something? :)

The only problem is, that I have absolutely no idea how the shape keys work in FBX (I only have some experience with skinned meshes). But - in the worst case scenario - you can always workaround this issue by exporting shape keys as different models, right? :)

[OniDaito]

Welcome to the last 4 days of Hell! :P

http://www.youtube.com/watch?v=gaEPj4GBBWc

Basically, FBX is really the only way to get stuff out of Max properly and its a pain! >< So far I've not managed it, and three.js has quite a few limits imposed on its skinning. That said, Im not far off a basic solution

[mrdoob]

@OniDaito are you doing a FBXLoader?

[OniDaito]

Not quite, I prefer writing stuff in Python, so I'm taking an FBX File and exporting json in the skinning format. Its close but no cigar yet.

[apendua]

@OniDaito Python you say... I'm just curious if are you using the FBX SDK for Python - the one provided by Autodesk - to read the file, or maybe some other solution? :)

[alteredq]

Skinning is always such a mess, there are just too many ways how to do it. Having it working for one case doesn't mean it'll work somewhere else.

For example I tried @apendua's latest changes to Blender exporter with few models that used to work and they weren't working anymore :S

[OniDaito]

@apendua Yes, I'm using the Python SDK. So far, I've managed to make 'something' reasonable but yes, agreed, skinning is a pain. I've made it work with Cinder and the FBX before now but that used Matrices directly. I dont know how to get it into Three.js format

[apendua]

@alteredq Have you tried the latest version from my blender branch? If you could be more specific about the errors you get, than maybe I will be able to fix them...

Speaking about the blender exporter, would you mind if I create a separate issue on this topic? I've at least one important question / proposition to state that is not directly related to the skinning issue and it may require a longer discussion :)

@OniDaito I'm afraid the biggest problem about working with bones is that - as @alteredq suggested - there are several ways to represent the skeleton and that we too often forget to state precisely which one we actually use. Concerning you last question about the pos, scl, rot, qrot attributes. As far as I now they are representing the bone's transformation relative to it's parent, so not global but local transform. The rot attribute may be ignored at the moment - it's kind of deprecated I think, use qrot instead - just look at the code of SkinnedMesh.js, @alteredq am I right?

Also note, that when you export the animation to three.js your keyframes should describe the skeleton pose in the same format as above (the skeleton rest position). In other words, in the three.js format the bones are being animated relative to their parents (in the parent's coordinate system).

[OniDaito]

Yes, I looked at the code and plotted the bones of existing meshes and skins and noticed they were local. I think I've got the hang of the three.js format now. One thing I did note was that all vertices MUST have EXACTLY 2 weights. Ive seen FBX files with vertices affected by 5 or more bones. This means rigid skinning is more the norm if going from 3DS Max. I think thats all there is to note on the three.js file format. Sadly, the FBX format is not so nice.

[alteredq]

@alteredq Have you tried the latest version from my blender branch? If you could be more specific about the errors you get, than maybe I will be able to fix them...

It just didn't do anything - no funny broken models but no animations either.

For some models, numbers were there in JSON and on sight looked normal but starting animation didn't do any movements.

For some other models there were just zeroes in JSON for skinWeights and skinIndices.

Also seems like with this new version material info was lost somewhere: instead of having multiple materials, everything was collapsed into a single dummy material.

The rot attribute may be ignored at the moment - it's kind of deprecated I think, use qrot instead - just look at the code of SkinnedMesh.js, @alteredq am I right?

Yup, I thinks so, just quaternions are used.

[apendua]

@OniDaito Doing more than 2 (but less than 4) influences in three.js should be quite easy now. However, it still requires some major decisions to make, for example:

1. In the file format, adding influnces attribute should do the job - but all importers will need to be updated :(
2. Do we allow influences=3? or just 1, 2, 4? (the powers of 2 can be a little more efficient from the shader point of view, right?

@OniDaito 5 influences is definitely insane :) but I think you can safely assume that we will have 1,2 or 4 in three.js in the nearest future :)

[alteredq]

One thing I did note was that all vertices MUST have EXACTLY 2 weights. Ive seen FBX files with vertices affected by 5 or more bones.

In Blender what we do is order bones by weights and take 2 strongest ones. In most cases this produces kinda ok looking results. I tried to dump numbers and these extra weights beyond 2 are usually very small anyways.

[apendua]

For some models, numbers were there in JSON and on sight looked normal but starting animation didn't do any movements.

@alteredq But the keyframes where there in *.js, right? Since, I only export the action which is attached to the armature... the old exporter just took bpy.data.actions[0], so it would export it under no conditions.

For some other models there were just zeroes in JSON for skinWeights and skinIndices.

This one is simple... you just need to parent your object to the corresponding armature and set the parent type to Armature. No support for armature modifier yet :( my bad...

Also seems like with this new version material info was lost somewhere

This is funny, since I never messed with materials... at least i think I haven't :) I would need a concrete example to investigate this one.

[alteredq]

@alteredq But the keyframes where there in *.js, right? Since, I only export the action which is attached to the armature... the old exporter just took bpy.data.actions[0], so it would export it under no conditions.

Yup, keyframes were in JS file

This one is simple... you just need to parent your object to the corresponding armature and set the parent type to Armature. No support for armature modifier yet :( my bad...

Cool, this fixed it. Also I got materials and animation working with that model ;)

This is funny, since I never messed with materials... at least i think I haven't :) I would need a concrete example to investigate this one.

It's this one:

https://dl.dropbox.com/u/26786471/tmp/models/cyberdemon.blend

I think materials here were my fault, I didn't check "export all meshes", but model still refuses to move.

Also exported JS file is notably smaller than it used to be with the old version of the exporter (0.5 MB vs 1.2 MB), so maybe some info is just not getting there?

[apendua]

@alteredq Fixed :) The animation should work now... the reason why the model didn't move at all was that the action curves in your .blend file (pretty cool model BTW) are not grouped by the corresponding bones names (as it is usually done, but not mandatory, in blender). Apparently, in the "old exporter", the names of these groups were needed to find the matching between bones and curves. So, are you sure that the animation used to work in the old exporter?

[alteredq]

@alteredq Fixed :) The animation should work now...

Great, thanks ;)

pretty cool model BTW

That's from Doom 3, it's that big guy here:

http://alteredqualia.com/three/examples/webgl_animation_skinning_doom3.html

So, are you sure that the animation used to work in the old exporter?

Good that you mention it. Yes, it was working, I fixed this in the "old exporter".

https://github.com/mrdoob/three.js/commit/ece0e5b315c8e0ff161e9460cad69c50a645d16b

But seems you based your new version of skinning exporter on some not completely up-to-date version of Blender exporter.

For example I also noticed your version was generating 3 version number string. I changed it to 3.1 already quite some time ago.

It would be good if you could try to merge your version with what's in dev now.

[apendua]

But seems you based your new version of skinning exporter on some not completely up-to-date version of Blender exporter.

@alteredq Yeap, that's correct. I was working with the master branch. Fine then... I'll try to merge against dev as you suggested.

[alteredq]

Always a good idea to work against dev branch ;) Otherwise chaos ensues.

Even in master though that version number is already there (also that fix):

https://github.com/mrdoob/three.js/blob/master/utils/exporters/blender/2.63/scripts/addons/io_mesh_threejs/export_threejs.py

Edit: it's weird, according to GitHub network view, you should have more or less fresh version from master branch, but then in this commit all hell broke lose: https://github.com/apendua/three.js/commit/87cfdf631696b5950c286e87583c8be19fc0eaea and diff just shows like if everything changed at once.

[apendua]

@alteredq I think it's because I'm currently having some problems with crlf EOL conventions :(

Also, I don't exactly now how to create a feature branch that will be easy to merge with mrdoob/dev. Any time, I try to create a pull request, I get a pretty huge list of commits, which I don't actually want to include. Is there an easy way, to restore my dev branch to be an exact copy of @mrdoob's dev? Since I already have altered it a little, I'm afraid that git pull won't do the job, right?

[apendua]

@alteredq Ok, never mind, I think I've found the answer myself... but the question about crlf remains actual :)