Working on SkeletonAnimation - JSONFormat

[codeart1st]

Hey guys, I'm working on SkeletonAnimation, but I don't understand how the bones will stored in JSONFormat.

{
  parent: BXindex,  // -1 for root bone
  pos: [x,y,z],
  rotq: [x,y,z,w], // quaternion
  scl: [x,y,z]
}

To understand it, I started to present the bones as THREE.Lines.

http://www.fiesta-library.bplaced.net/test/webgl_animation_skinning.html

But something is wrong with that.

In the JSONFormat is only one position for each bone. In Blender bones have two positions - tail and head. So how works the Three.js Bones?

[codeart1st]

Nobody here who knows how the bones in Three.js works?

[mrdoob]

Nope :( It was initially implemented by @empaempa but we haven't been able to clean it up just yet.

[codeart1st]

Now I understand how the bones will stored in JSONFormat. Look at the example for it. http://www.fiesta-library.bplaced.net/test/webgl_animation_skinning.html But I have some problems with the quaternions. I don't understand how quaternions rotates vertices - the mathematically background. The "rotq - quaternion" in the file format is presented locally to his parent. How can i rotate vertices relative to its parent with these quaternions?

[empaempa]

Hey! As @mrdoob points out, the skinning was implemented way back (more or less as a test when we did ROME) but not really finished - so there might be bugs all over the place. Hope not ;)

Before rendering, the quaternions are converted to Matrix4 using Matrix4.setRotationFromQuaternion in Object3D.updateMatrix. These Matrix4 transforms the vertices in the vertex shader. We use quaternions for animation because they are safe to interpolate between when you do animations (matrices and euler angles have edge cases where they fail to interpolate in a way that is visually satisfying). Your animation exporter should be able to output the quaternions for each key frame.

The easiest way to rotate quaternions is using the THREE.Quaternion.slerp( fromQuaternion, toQuaternion, destinationQuaternion, timeBetweenZeroAndOne ). This is used in the THREE.Animation.update().

All this said, you can parse the bone-hierarchy and set .useQuaternion to false and rotate the bones using eulers.

Hope you get it to work :)

[codeart1st]

First of all, it seems that bone-, skin- and weight-export now works perfectly for my model (not tested with other models at the moment). Also I work on the animation-export - first only with one keyframe for testing. The animation-export mostly works, but I have still problems with the quaternion rotation in animation-export, so i hope that your comment can help me.

[empaempa]

If I where you, I wouldn't trust THREE.Animation.update to do the right thing. Maybe try to write a simple test case where you rotate one of your bones using euler angles and then take it from there. Use an uncompressed version of Three and the debugger ;D

[codeart1st]

http://www.fiesta-library.bplaced.net/test/skeleton.html In the example use mousescroll to zoom in and out and use the pressed mouse key to change the camera. The blue planes are the bone positions, the black lines are the normal parent relationships and the red lines are only joints with no functions. The reason for this is, that my normal model in blender has 45 bones and every bone has a head and a tail - and not every tail has the same position of the parent head in the most models - there are some seperators (red lines). Three.js Bones have only on position and the second position is like the parent bone position, so I exported for every bone in blender to Three.js Bones (totally 90 bones). It seems that it works good, but I have now the double count of bones and I think the double performance consumption. So is it a good solution or not.

[empaempa]

Cool, you're getting there :)

As you say, THREE.Bone has only one position and rotation (or one matrix, let's call it the "head") as really nothing else is needed. If you like, you can imagine the bone's "tail" to be somewhere offseted along the z-axis of the head's matrix. Usually 3D programs displays bones with head and tail, as it feels more natural and intuitive to work with, but really it's just a way to rotate the head.

At THREE.SkinnedMesh construct the model is "posed". This is a process to "reset" the vertices to its influencer(s) initial rotation. Let's say your model's elbow is rotated 45 degrees at frame zero, hopefully the elbow's matrix is rotated 45 degrees as well. (The pose function also do some optimizations (projecting them into a local space etc) to save multiplications in the vertex shader.) Not sure why I'm mentioning this, but it might help to know ;)

45 bones is pushing it. I'm surprised it even works, you should run out of uniforms...

Keep it up!

[codeart1st]

http://www.fiesta-library.bplaced.net/test/skeleton.html The first keyframe is now exported perfectly from blender 2.62. The skeletal shows only the position translation, because I had no bock to include the quaternions, but the model is now on the right pose. -> Next step: Export all keyframes. But still I need the double number of the THREE.Bones. I try it with the normal number, but there were a lot of artefacts and the final pose was misrepresented. I think it runs without the double number of bones, but then I should somehow recalculate all data from blender :/

Thanks @empaempa for your help :)

@empaempa To illustrate my problem with the bone exporting. You can look at this picture. http://www.fiesta-library.bplaced.net/test/Unbenannt.png Normally every bone head (big side) has the same position as the parent's tail (small side) position (e.g. look at the spines). And the quaternions in Blender rotates the tail in relation to the head position of the same bone (blue color). When I export the Blender bones to Three.js, every tail position is in relationship to his parent's tail - usually it works. But sometimes (the red color example) the Three.js bone is different to the Blender bone. And I have no idea how I can solve this issue without using the doulbe number of bones.

[codeart1st]

@mrdoob It's mysterious. I run the example from my comment before on my linux system with the chromium browser - only an Intel Atom CPU and a Intal GMA 954 onboard graphic card / very shit platform - and it works perfect without any error. Then I tried the example on a windows xp machine with a quadcore cpu and ati radeon graphic with firefox and chrome and the example didn't work anymore - an error with the WebGL vertex shader I think, but how? Error: "Could not initialise shader"

[empaempa]

It's really hard for me to say what's happening, but I suggest you start with a much simpler model. When we were working with this, we started with two bones that deformed a cylinder. And there was a lot of debugging and thinking to get it right, so you're not alone ;)

We used an entirely different 3D-package, so I can't really help you with the Blender specifics. However, given you know how matrices work, I think you should view the Blender bones as matrices placed at the head with the Z-axis is pointing towards its tail. And I don't think you should really bother about how their (head)position relates to their parent's tail position. It's not really interesting, from a mathematical POV.

About the shader compilation error, where there any specific details in the console?

[codeart1st]

Tommorow I watch which error it is in detail. Do you have the error too with the example above? I am convinced that I will find the right way to exported the bones correct next time ;) The animation works with the double number of bones, so I hope that I will get it with the original number of it... It needs a little bit more time, because I don't have any knowledge about Blender, Python and animation, but I feel happy with every small milestone :)

[empaempa]

You might have run out of uniforms, actually. Not sure if that renders a vertex shader compile error, but that's the first thing I'd check. No, don't get any error here :)

[codeart1st]

Could you explain me please, what do you mean with 'run out of uniforms'? I'm not a genius of graphic programming ;)

And tanks again for the great support =)

Edit: This message will be shown on my win7 laptop and on my win xp desktop pc, but works with a bad linux system. Could not initialise shader VALIDATE_STATUS: false, gl error [0] initMaterial WebGL: INVALID_OPERATION: getUniformLocation: progam not linked

Error only with the double bone number example. The 45 bones example works with the exception of the wrong pose.

[empaempa]

Yes, then you've run out of uniforms, as I suspected.

Uniforms are variables you send to the shader - and there's a maximum amount for how many the vertex shader can take and how many the fragment shader can take, which varies depending on your hardware. I think each bone takes four vec4-uniforms (a matrix4, basically). If you have 90 bones, that's 90*4=360 uniforms. Then you add uniforms for lighting, texturing and whatnot on top of that. Probably your hardware only supports 254 vertex shader uniforms, which is the most common limit:

http://webglstats.com/

You can verify this using the WebGL Inspector (under the State-tab):

http://benvanik.github.com/WebGL-Inspector/

My MacBook pro supports 1024, which is enough to run your demo. This is basically the best reason to why you should try and remove the tail bone. One thing to look up: do the tail bone influence any vertices? If yes, try to remap them to be influenced by the head bone. Also, I hope you're aware that right now the amount of influences per vertex is two bones. This can be increased, but requires some changes to Three.

[codeart1st]

Wow, now I know what you mean with the uniforms. Thanks alot. http://webglstats.com/ This is a nice WebTool and I'm surprised that my linux system supports 4096 uniforms and windows only 254. No in blender the full bone influence the vertices. The head of the bone is like a parent joint in Three.js and the tail is relative to this head/parent joint. And yes, my models only need 0-2 influences per vertex, so it's ok for Three.js. http://www.fiesta-library.bplaced.net/test/skeleton%20(Kopie).html Here's the version with 45 bones -> position transform looks nice, but working on rotation -> some bugs in it

[codeart1st]

Omfg, it's done. Only I have some clean-up to do, because in some cases it's a little bit hard-coded ;) Now 45 bones get it to work. Next step: Export all frames with the right frame-step-option. I tested it a bit and I think there is a small bug in Animation.js with the detection of loop. I will write about the progress later...

Nice jop @empaempa

[empaempa]

Yay! And I'm not surprised about the bug in Animation.js. It's so not tested. Please post a link when there's something moving around :D

Go go go!

[codeart1st]

www.fiesta-library.bplaced.net/test/skeletal.html www.fiesta-library.bplaced.net/test/skeletal2.html

Two examples with different actions. More frames later ;)

[mrdoob]

Awesome! :)

[codeart1st]

www.fiesta-library.bplaced.net/test/test.html

All frames will exported now. But on all my PCs Animation.js seems to work wrong with looping - also with the buffalo.js model. Can everyone look at it, because I have problems to understand the code.

[codeart1st]

@empaempa Testing the looping algorithm and it works good, all 2,3 seconds I get a message in my console, that it is looped now. Also I tested all interpolation types and with every type the problem wasn't solved.

console.log( "THREE.Animation.update: Warning! Scale out of bounds:" + scale + " on bone " + h ); There are often scale errors with my models, also with the buffalo model. Could this be the problem?

[empaempa]

It could be. If I were you I'd make a VERY simple 2-bones object with a very simple animation - like a cylinder, wagging back and forth with like 2-3 keyframes. This way you have a very limited amount of data to debug, and easier to nail what's happening. I'd also comment out everything that has with JITCompile to do, as this is merely an optimization (a quite premature optimization, too (it's saving matrices for each frame it has computed, so you don't need to recalculate them next time)).

Stick to linear interpolation, to start with. Not sure the other interpolations work.

The theory goes: take a time, find previous and next keyframe for position, rotation and scale (each frame doesn't need all values to be keyed). Interpolate between these two keyframes and put together a matrix. When looping, you're possibly looking at having the first keyframe (at time 0) as the "next" keyframe.

Unfortunately I'm busy today and leaves for 4 weeks vacation so I've got very limited time to help out with the actual code. I'll try to help you here, in this thread, as much as I can :)

[codeart1st]

It's done. Sorry, that I asked faster than I think about this self ;) So the problem was, that the JITCompile, which is by default set on true, works wrong with the models. Is JITCompile important and what is it in detail? - I don't know it. What I know is, that when is set to false, it works brilliant. Thanks guys!

[empaempa]

Great! See my comment above about the JITCompile. Had a feeling it was this that wasn't working properly :)

[codeart1st]

I will make a performance test in the next hour, because I think the performance is good enough without JITCompile at the moment. I will upload an example later. Now Three.js has a great skeletal support :D

[codeart1st]

In my opinion it's good enough without using of JITCompile. Here are two examples:

www.fiesta-library.bplaced.net/test/example.html 277,1 kB json-file size - 1 sec length animation - walking

www.fiesta-library.bplaced.net/test/example2.html Performance-Test: 1,6 MB json-file size - 25 sec length animation - dancing

FPS are good on a Ubuntu/Linux PC with Intel Atom N230 and Intel GMA 945 Graphic Adapter. I think it's done?

[empaempa]

Awesome! Works really good on my 2.5-year old MacBook pro, too.

The JITCompile should increase animation update performance by at lot. The theory goes that you save all the matrices you calculate to a cache, so next time you come to that frame you simply take the cached matrix instead of interpolating quaternions and convert them into matrices. It should be pretty easy to implement, but obviously something is buggy :)

[codeart1st]

I think that's not my work at the moment ;D

[mrdoob]

Great! See my comment above about the JITCompile. Had a feeling it was this that wasn't working properly :)

Premature Optimization™ - @alteredq

;)

[mrdoob]

www.fiesta-library.bplaced.net/test/example.html Performance-Test: 1,6 MB json-file size - 25 sec length animation - dancing

Wrong link?

[codeart1st]

Yes, you're right :D I forget a '2' in the link. The right link is now in the link-comment.

[mrdoob]

That's pretty cool! :D

[alteredq]

Really nice indeed ;)

[empaempa]

Great stuff!

[codeart1st]

www.fiesta-library.bplaced.net/test/2.62.zip

Here is the exporter code. Extract the parts which you need. It's an adjusted version for my project without import features and scene export features. No time to make a patch, because it's time for my own project.

Thanks guys for the support. Hope that the code from me will improved in the feature ;)

[mrdoob]

@n3tfr34k have you modelled and animated that dancing guy? would be great if we could use that as skinnedmesh example isntead of the buffallos that we have right now. would you be up for "donating" it to the project? we can add a credit link to whatever page you want.

[codeart1st]

Sorry, this is a good idea, but this wasn't my models. I imported the nif-files from an online game to blender for it. :/

[alteredq]

@n3tfr34k Could you put blend file for that dancing guy model somewhere, not to be used in the example, just to have something to test with for merging?

I transplanted your code from ZIP file into latest Blender 2.63 exporter but it fails with a random animated skinned model I tried (exported geometry isn't in proper base T-pose, just some animation pose, skeleton isn't rotated correctly and most strangely skeleton is larger than the mesh, all of these resulting in completely messed up animation).

I also tried your exact code from the ZIP with portable Blender 2.62 and it didn't work either on that model (similar issues like merged code in 2.63 exporter).

BTW how did you import NIF files into Blender 2.62? I could only find NIF tools for up till Blender 2.49 :/

http://sourceforge.net/projects/niftools/files/blender_nif_scripts/2.5.x/

[codeart1st]

www.fiesta-library.bplaced.net/test/Fig-m_Bip01_Walk_One.blend www.fiesta-library.bplaced.net/test/Body.png www.fiesta-library.bplaced.net/test/Legs.png www.fiesta-library.bplaced.net/test/Shoes.png

Walking is better for starting (no head, because it's also simpler for starting). Yes it could be that the exporter need a little bit improvement for other models - I tested it only with my models. Good Luck ;)

Importing with Blender 2.49 -> deleting all unimportant parts -> storing to .blend -> openinig with Blender 2.6+ (:

Edit: I think skeletal exporting with Blender confuses all of us :D

[alteredq]

Thanks. This one seems to be working correctly (exported from merged Blender 2.63 exporter), so that's a good start ;)

Curiously, for a change, with this model I'm not able to see any animation in Blender but exported JSON works ok.

Importing with Blender 2.49 -> deleting all unimportant parts -> storing to .blend -> openinig with Blender 2.6+ (:

Aha, no free lunch with NIF files :/

Edit: I think skeletal exporting with Blender confuses all of us :D

Indeed ;) At least you broke the initial barrier, now it can be incrementally improved upon.

[codeart1st]

The reason why you don't see the animation in Blender is that you need to change the armature options from rest-option to pose-option and every model which you want to export need rest-option before exporting.

[alteredq]

Thanks, that did the trick.

Finally I managed to export the model that was causing troubles - no changes to the exporter, I just messed with the model in Blender (there was extra transform in armature object).

BTW I think there may be still something wrong with looping playback, even without JITCompile caching.

There is some glitch at loop boundaries, looks like transition between last and first frames is not interpolated. This is more visible with bigger difference between first and last poses.

[codeart1st]

Ok, nice to read. It's always nice to read that the written code works fine. Yea, Animation.js need some changes.

[empaempa]

@alteredq If you like, I can ask Zoink if they have some model+animation that you can use. They're using Max, but I guess there's some export-format that can be imported into Blender.

[alteredq]

@alteredq If you like, I can ask Zoink if they have some model+animation that you can use.

That would be awesome ;) They know how to create proper models for real-time (in addition to making them pretty).

I'm went through many random models available from the web and while some of them look great, most of them are in terrible shape and do not survive conversions :S.

They're using Max, but I guess there's some export-format that can be imported into Blender.

Not much experience with this but there is always Collada. That's how I made these morphs (small ones are Collada => Blender => JSON, big one is directly Collada):

http://mrdoob.github.com/three.js/examples/webgl_loader_json_blender.html

[codeart1st]

Nice progress @alteredq . Good news in dev branch :)

[empaempa]

And some more good news: Zoink will supply model+animation as soon as... I don't know, things move slowly during the summer ;)

Of course they like to be credited and linked to, but I guess that's standard procedure.

Now I need to head to the beach and take a swim!

[alteredq]

And some more good news: Zoink will supply model+animation as soon as... I don't know, things move slowly during the summer ;)

Cool cool.

Of course they like to be credited and linked to, but I guess that's standard procedure.

Yup. Being in three.js example is usually a good publicity ;)

[apendua]

Hi guys,

Great job so far on the exporter :)

As @alteredq suggested (another thread), I looked into the current code. I'm currently working on a project, where I need some more functionalities than the current version has to offer, so I used your code as a starting point and tried to implement them by myself... and I think it is kind of working. If you're interested, the code will be available here:

http://oniostudio.com/io_mesh_threejs

I've just finished working on exporting actions to a separated file (suffixed: ".actions.js", "All actions" toggle in the exporter menu). I also made some modifications to the way the bone transforms are stored into the file. I can see, that the development version of the exporter uses a kind of trick by putting identity rotations on all bones in the skeleton structure. This clearly implies that also animation keyframes need to be transformed properly before they're saved into the file, and in my opinion it complicates everything a lot. Is there a reason for doing that, or maybe is it just a workaround of some kind? I tried to work on this issue for some time, and managed to get to the point where basically the skeleton structure (original bone transforms) remains untouched (modulo the parent "head-tail" issue), and the keyframes are modified only a little (i.e. transformed from local space - blender flavor - to the parent's space).

Some other changes includes:

1. The mesh <-> armature correspondence reflects the blender parent relations, i.e. if an object is parented to an armature object in the "ARMATURE" mode (not "OBJECT"), then the bones will be generated from this parent armature.
2. If the armature (to which an objects is parented) has some animation_data on it, i.e. some action is attached to it (check blender action editor), then these action will be exported in the form of "animation" chunk in the mesh file.
3. If the "All actions" toggle is on, all actions from the .blend file will be exported to a separate file (mesh base file name + '.action.js'). Missing channels are field with default values (bone transforms in the rest position), unnecessary channels (no bone with corresponding name) are ignored.