removeBoneFromList - potential error

[Scraft]

Hi,

I ported the project to C#/Unity, it seems to run correctly, albeit very slow (around 5-6FPS if solving every frame, on a high-end laptop).

One line I had to comment out is in bold below:

public void removeFromBoneList(AbstractBone abstractBone) { if(bones.contains(abstractBone)) { bones.remove(abstractBone); boneMap.remove(abstractBone); } }

Is this line correct? It appears to be attempting to move an element from a HashMap by value not key, which I presume will not do anything? I do very little Java, so I maybe missing something.

Out of interest, what sort of performance would you expect to get when solving every frame (I have been trying ConstraintExample_Kusudama and SimpleExample as my tests). I don't know if performance issues have been introduced in the Java -> C# process, or whether this library is meant more for offline calculations than runtime (using it for things like working out where a human skeleton is based on three pieces of tracking data for head and both hands).

[EGjoni]

The line you identified is a bug, yes (good catch!). However, that function is for user convenience and (if I recall correctly) not called internally, so your modification probably shouldn't be affecting performance.

The library was created for an animation program, so its default settings prefer quality and consistency first, and speed second (aiming for interactive rather than real time). That said it should be usable for real time applications where processor cycles aren't at a premium (or you're willing to sacrifice quality). If you're porting this specifically for use in a one-time project, and don't care so much about efficiency beyond your particular project, then here are the lowest effort + high reward things you'll want to take a look at:

1. What value have you assigned to the iterations parameter? Depending on how your armature is set up, there is at the very least a linear relationship between the number of iterations per solution and the number of solutions per second. So if you need the solver to respond 5 times as fast, try setting the "iterations" parameter to a fifth of your current value.

2. However, decreasing the number of iterations has the drawback that your results are less likely to converge by the time the solver has finished, which might result in bones that seem to chase (instead of track) fast moving targets. This can be avoided by increasing the "dampening" parameter. The dampening parameter establishes a hard limit on how many degrees the bones are allowed to rotate per iteration. The larger the dampening parameter, the fewer iterations will be required for the solver to converge. However, the higher this parameter is set, the less evenly distributed / graceful the solution will be (particularly, more of the rotation will occur in the outer bones, and less in the inner bones; so setting this parameter too high will result in situations where the armature will, for example, move the arms towards the targets, but not bother to include rotation in the spine if it can get away with it. This can lead to somewhat robotic results). How much grace you're willing to sacrifice for speed is a judgement call that you'll have to evaluate empirically for your use case.

In short, your best bet for attaining the biggest performance increase with the least additional effort is to first relax (increase) your dampening parameter to the largest value that still yields subjectively acceptable pose quality. Then, having decided on that acceptable level, decrease the solver's "iterations" parameter until you achieve the minimum acceptable level of chasing (frames where the solver hasn't quite reached the target). If at the end of this you find you have performance to spare, then great! You can use that to bump up whichever of the two (pose quality or convergence rate) you care more about.

If on the other hand you find that you end up in a situation of mutual exclusivity between pose quality and convergence rate, or if performance is at a premium such that spending fewer cycles in the solver would allow you to spend more elsewhere, then here are some additional optimization details to consider:

3. How are you calling the solver? Once per frame? If so, you can likely increase apparent convergence rate by accounting for the peculiarities of realtime use cases. Particularly, try setting the iteration parameter to an extremely low value (ideally 1, but maybe a bit higher depending on implementation specific overhead), then set the rest of your code up so that you call the solver on a loop (perhaps in its own dedicated thread) which runs regardless of whether or not a new frame is available. This way, the solver will not waste a bunch of iterations trying to converge on the targets of an old frame when there is a new and more recent frame available that it should be trying to converge to, and as a bonus, instead of waiting for the next frame when it's done, it can can spend that additional time on another iteration to further converge on the available frame. This approach is definitely more ideal for realtime use, however, the version of the code you've ported might require you to work around potential concurrency issues if you try to read the armature's pose while the solver is still working on it (I can't remember if the last version I pushed includes changes that alleviate the need for this). A trivial, low-effort hack around possible concurrency issues might be to create two copies of the armature: one on which you run the solver, and one which mimics the pose of the solved armature after every solution.
4. The tranquil solver and ambitious solver have somewhat different performance characteristics, but I believe the tranquil solver should be less prone to concurrency issues if you want to minimize workarounds.
5. Use floats instead of doubles if you aren't already (though that might lead to instability, haven't tested).

I suspect that suggestions 2 and 3 should be enough to address your performance issues. If your test is much faster in the java / processing demo, and presuming there isn't any peculiar weird thing in C# or your port, then I suspect 3 is the culprit.

If the java version of your armature is just as slow for you as your port, then I will have to look at the code to make sure I didn't accidentally leave some debugging procedures in or something.

All of that being said, I have expanded this library significantly since I last pushed it, but have been neglecting updates. If things like target orientation (instead of just position) are important to you as well, and you're willing to port another few classes / changes, I can clean up and push the latest version (which also has some potential efficiency improvements, particularly in backing the scene graph by actual matrices, thereby opening up the potential for SIMD optimizations when porting (or reusing existing scene graph implementations in Unity or whatever)).

Note also that while this library should allow for real time use, the way it's set up is going to be a significant CPU resource hog. So games in the wild are kind of a poor fit for the current structure. But this is more a byproduct of the use cases it targets than fundamental limitations to the algorithms its backed by, so if you're interested in making this a plugin or something, it might be more appropriate to implement the concepts rather than the code. I've been meaning to write a blog post exploring / explaining how it works, so let me know if that's something you're looking for and I'll take it as a hint from the universe that I should move it up the priority queue.

[Scraft]

Thank you for your detailed response.

1. What value have you assigned to the iterations parameter? Depending on how your armature is set up, there is at the very least a linear relationship between the number of iterations per solution and the number of solutions per second. So if you need the solver to respond 5 times as fast, try setting the "iterations" parameter to a fifth of your current value.

I have been using the defaults in the sample, which seems to be 0.1/50 during init, and then 0.01/25 during each update. As a reference point, in a 'basic' FABRIK solver I have experimented with, I was using 5 iterations, so that does line up with what you are suggesting.

2. Dampening

Thank you for explaining how this works, based on this I can certainly tweak this parameter.

3. How I update

I am calling the solver once per frame, in my test bed (basically your samples) I have changed it so rather than clicking a point and then letting it solve, it solves every frame and I can real-time adjust the position of the rootBone pin. If I can get to a nice, stable solution, looking at threading is certainly a possibility.

4. Which solver

I am currently using the tranquil and must admit I haven't experimented much here yet

5. Use floats

So this was my first go-to for trying to improve performance, and it did seem to yield a small win (although on some test runs the win sometimes was close to zero). This did not put me off, as, with Unity, whilst user scripts are C#, the backend is optimized native code. Unity works with floats, not double, so once I had a floating point version working, I was able to substitute out the DVector and PVector classes for the Unity native Vector3 class. This has yielded a vast performance win, the stats look something like:

• 52 FPS - Unity backed float version
• 13 FPS - standard c# float version
• 11 FPS - direct Java -> C# port (using doubles)

There is still a lot of math taking place within the solver, some of which potentially could be replaced with Unity native math (perhaps part of the Rot class could be more directly replaced with Quaternions).

If things like target orientation (instead of just position) are important to you

That would be fantastic. So, my use case, if you are wearing a Virtual Reality headset, and have two Virtual Reality controllers in your hands, I can get accurate positional and rotation data back for each. I would like to feed that data into Everything-Will-Be-IK and get back a predicted position of the skeleton. I have been going through a through different solvers for this and stumbled upon Everything-Will-Be-IK and from the video, it looked like it had more power than other simple implementations (in particular with regards to the constraints).

Now, onto the bad news, although I have made progress with the float version, and whilst I believe it is doing the right thing (it is completely possible there are bugs of course) it does feel quite unstable. For reference, here is the straight port (double):

https://youtu.be/Q8lcaImdsH0

Here is the float version:

https://youtu.be/uOkqoLDD87w

If you have any thoughts about the above it would be good to hear (i.e. do you feel the above is caused by float vs double issues, or do you feel there are some bugs in the float version). Also, one function I wasn't sure how to port the best way is:

public static float linearCombination(float a1_, float b1_, float a2_, float b2_)
    {
        double a1 = (double)a1_;
        double a2 = (double)a2_;
        double b1 = (double)b1_;
        double b2 = (double)b2_;

        // the code below is split in many additions/subtractions that may
        // appear redundant. However, they should NOT be simplified, as they
        // use IEEE754 floating point arithmetic rounding properties.
        // The variable naming conventions are that xyzHigh contains the most significant
        // bits of xyz and xyzLow contains its least significant bits. So theoretically
        // xyz is the sum xyzHigh + xyzLow, but in many cases below, this sum cannot
        // be represented in only one double precision number so we preserve two numbers
        // to hold it as long as we can, combining the high and low order bits together
        // only at the end, after cancellation may have occurred on high order bits

        // split a1 and b1 as one 26 bits number and one 27 bits number
        double a1High = System.BitConverter.Int64BitsToDouble(System.BitConverter.DoubleToInt64Bits(a1) & ((-1L) << 27));
        double a1Low = a1 - a1High;
        double b1High = System.BitConverter.Int64BitsToDouble(System.BitConverter.DoubleToInt64Bits(b1) & ((-1L) << 27));
        double b1Low = b1 - b1High;

        // accurate multiplication a1 * b1
        double prod1High = a1 * b1;
        double prod1Low = a1Low * b1Low - (((prod1High - a1High * b1High) - a1Low * b1High) - a1High * b1Low);

        // split a2 and b2 as one 26 bits number and one 27 bits number
        double a2High = System.BitConverter.Int64BitsToDouble(System.BitConverter.DoubleToInt64Bits(a2) & ((-1L) << 27));
        double a2Low = a2 - a2High;
        double b2High = System.BitConverter.Int64BitsToDouble(System.BitConverter.DoubleToInt64Bits(b2) & ((-1L) << 27));
        double b2Low = b2 - b2High;

        // accurate multiplication a2 * b2
        double prod2High = a2 * b2;
        double prod2Low = a2Low * b2Low - (((prod2High - a2High * b2High) - a2Low * b2High) - a2High * b2Low);

        // accurate addition a1 * b1 + a2 * b2
        double s12High = prod1High + prod2High;
        double s12Prime = s12High - prod2High;
        double s12Low = (prod2High - (s12High - s12Prime)) + (prod1High - s12Prime);

        // final rounding, s12 may have suffered many cancellations, we try
        // to recover some bits from the extra words we have saved up to now
        double result = s12High + (prod1Low + prod2Low + s12Low);

        if (double.IsNaN(result))
        {
            // either we have split infinite numbers or some coefficients were NaNs,
            // just rely on the naive implementation and let IEEE754 handle this
            result = a1 * b1 + a2 * b2;
        }

        return (float)result;
    }

At the moment, I am just converting to and from doubles for this function, but as this function is designed to allow more stability/accuracy, I wondered whether it could be an initial problem area. I wondered whether this function could be updated to work with floats, there is information about the method being used here:

http://www.ti3.tu-harburg.de/paper/rump/OgRuOi05.pdf

But it wasn't clear to me how it would be adapted to floats.

My hope is somehow there is a way to improve the stability of the float version so it is useable, as I feel if this was the case, there would be a path to get the performance acceptable for my use case.

[EGjoni]

Oh wow yeah that is hilariously bad performance (especially for so simple an armature). If Unity has a profiler or something that lets you look at how much time is spent in each function, I would be very interested to see where exactly it's choking.

It's good that using Unity backed floats results in such a performance win, but I think it might still be competing against a huge performance loss elsewhere. I say this primarily because even using doubles and no SIMD instructions on a considerably more complex armature than the one you linked to, the Java version is easily able to handle 1500+ iterations per second on a 6 year old core i5.

For reference, here is a video of the tranquil solver and its performance characteristics over a range of parameters as I tweak them on the left hand panel (note that the dampening value in the UI has been converted to degrees for display purposes, you should not actually set the dampening to a higher value than PI, as this amounts to no dampening. Also note that in this example, the solver is invoked once per mouse input event, not once per frame, but even so your port should still be getting much better performance than 53 fps).

There is still a lot of math taking place within the solver, some of which potentially could be replaced with Unity native math (perhaps part of the Rot class could be more directly replaced with Quaternions).

Rot is mostly just a wrapper around the Apache Commons Rotations class, which is itself essentially just a collection of Quaternion and Matrix convenience functions; so yes, wherever you find opportunity to use whatever Quaternion functions Unity offers, absolutely do. But as a heads up, do be mindful that some libraries use the convention q0=x, q1=y, q2=z, q3=w, while some others use q0=w, q1=x, q2=y, q3=z. (this can be a huge pain if you forget to swap a q3 for a q0 somewhere).

Now, onto the bad news, although I have made progress with the float version, and whilst I believe it is doing the right thing (it is completely possible there are bugs of course) it does feel quite unstable.

If both are using the tranquil solver with Kusudama constraints (instead of Euler constraints, which were added on a whim and aren't very well tested), then I'm inclined to suspect that the instability is due to how floating point errors are being handled, yes. Using the built in Unity quaternion implementation could help with this.

At the moment, I am just converting to and from doubles for this function

I think this is probably a very bad idea (given the IEEE754 reliance) and quite plausibly introducing further instability. If you can avoid that function altogether, and rely instead on some built in Unity or C# library to do a fancy precision linear combination, that would be ideal. Another option would be to replace

public Rot(DVector iv1, DVector iv2, DVector iu1, DVector iu2) {
        Vector3D v1 = new Vector3D((double) iv1.x, (double) iv1.y, (double) iv1.z);
        Vector3D v2 = new Vector3D((double) iv2.x, (double) iv2.y, (double) iv2.z);
        Vector3D u1 = new Vector3D((double) iu1.x, (double) iu1.y, (double) iu1.z);
        Vector3D u2 = new Vector3D((double) iu2.x, (double) iu2.y, (double) iu2.z);
        try {
            rotation = new Rotation(v1, v2, u1, u2);
        } catch(Exception e) {
            rotation = new Rotation(v1, 0f);
        }
}

To something that manually does the same thing in terms of two separate rotations. Like:

public Rot(Vector3 iv1, Vector3 iv2, Vector3 iu1, Vector3 iu2) {

    Rotation firstRotation = new Rotation(iv1, iu1); 
        firstRotation.normalize(); 
    Vector3 t_v2 = firstRotation.applyTo(iv2); 
    Rotation secondRotation = new Rotation(t_v2, iu2); 
    Rotation result = secondRotation.applyTo(firstRotation);  //composes the two rotations 
    rotation  =  result.normalize();            
} 

I believe the above constructor is the only spot calling the function that relies on the fancy linear combination implementation. So hopefully that change suffices.

I have been going through a through different solvers for this and stumbled upon Everything-Will-Be-IK and from the video, it looked like it had more power than other simple implementations (in particular with regards to the constraints).

Yup. Basically Inverse Kinematics is super easy right up until you have to worry about constraints. At which point, it is suddenly super hard.

My hope is somehow there is a way to improve the stability of the float version so it is useable, as I feel if this was the case, there would be a path to get the performance acceptable for my use case.

I feel like there isn't any inherent algorithmic reason to expect huge stability issues when going from double to float in your ultimate use case (they would be more likely in your test case though, depending on how your armature is constrained). In general. the nature of the two solvers is such that they should inherently correct for numeric instabilities whenever a solution is readily reachable. For a head and hands rig, a solution should pretty much always be readily reachable (I guess unless the player is a contortionist).

Beyond that, it's also possible that there is a bug in the version you're porting from which I have since inadvertently corrected. I'm working on cleaning up the new version's code now, so I guess we'll see. In the meantime, you wouldn't be wasting effort if you began / continued hooking things into Unity's quaternion implementation, as the new version still relies heavily on quaternion math.

[Scraft]

I did a profile, however, C# managed code is just really slow (under Unity at least), it is just a huge mess of boxing/unboxing values, indirect memory access to ensure no out of bounds issues are hit, etc. I have seen this exact issue in the past where maths code in a C# script performs 10X (or more) worse than the same code written natively, put in a DLL and called from the C# script. There is a large amount of Garbage Collection being generated and cleared up too.

I noticed whilst comparing different versions, that if I just left the float version (the version where all I have done is basically replace double's with float's, although the same issue also occurs in the version where I converted to Unity native types for Vectors etc) going for several minutes, it all just falls apart (you may want to speed YouTube up to double speed or more):

https://youtu.be/9bVnV_t69AU

I have combed through the changes between the double and float versions several times now, but I am failing to see any real difference, nearly all the changes are just things like:

• float -> double
• 1.0d -> 1.0f
• Math.Sin -> Mathf.Sin

When your new version hits, I'll basically convert that to C#, get that working, then do the same double -. float and then float -> Unity types and see what results I get. That will naturally replace anything I have looked at so far, that said, if you can suggest anything for me to look at, from the above video, I am happy to take a look. As I fear the above issue is going to come back with any float version until a diagnosis can be made of what is causing it. FWIW, I changed the linear combination function to just do a1 * b1 + a2 * b2 and still get the issue, so that isn't the code causing the issue.

Your video looks great and represents exactly what I would like to achieve, a really stable, robust approach to IK, which maintains target position and orientation whilst obeying a set of well-defined constraints. It is also exciting to think it should be possible to get decent speed ultimately (I will worry more about speed once I have got a float version robust).

[Scraft]

Oh: a little question, in terms of this C#/Unity version, I am more than happy to put the source on GitHub, would it make the most sense for it to be a new project (EWBIK-C#), or a fork of this project with some branches (branches/c#-double branches/c#-float branches/c#-unity-native-types) or some other variations?

[Scraft]

I realize I didn't answer one part of the question, which is to say my faster version is using UnityEngine.Vector3 and UnityEngine.Quaternion as direct substitutes for what was the DVector/PVector and Quaternion classes. I did indeed run into the issue of Unity having quaternions set up so the first three elements are the XYZ and the forth is the W, however I was lucky that this was actually easier than I had expected it to be to work around, with some getQ0(), getQ1(), getQ2(), getQ3() methods returning the right thing, and being careful on the order of parameters to the constructor.

So, I have three versions, the straight port (double) version, the conversion to float (singles) and the conversion (from the float version) to the version which uses Unity.Vector3 and Unity.Quaternion. The double version seems to behave correctly, the float and Unity.Vector3/Unity.Quaternion versions behave the same and is captured in the above video where things just fall apart. If I move the target around, the skeleton updates and attempts to follow, and gives the impression it is generally working relatively well, but something somewhere is causing this constant issue which completely destroys the implementation. Hopefully, that is all clear.

[EGjoni]

I have seen this exact issue in the past where maths code in a C# script performs 10X (or more) worse than the same code written natively, put in a DLL and called from the C# script. There is a large amount of Garbage Collection being generated and cleared up too.

Ah, well that solves that mystery. Though, with regard to the Garbage Collection, this code's current but somewhat inconsistent philosophy is "whatever, that's what the garbage collector is for." So optimizations are possible.

That will naturally replace anything I have looked at so far, that said, if you can suggest anything for me to look at, from the above video, I am happy to take a look.

Nope! The video you just posted is perfect. I've seen that drift issue before and I believe I know how to fix it. The only thing I'm unsure of is if it will still need fixing in the new version. (The solution is to just orthonormalize each bone's local axes every time it gets touched by iterateCCDStrand(), and possibly forcibly translate each child back to the parent bone tip [and I can give you more detailed instructions if you'd prefer to try it ahead of time rather than just wait for the new version]).

Oh: a little question, in terms of this C#/Unity version, I am more than happy to put the source on GitHub, would it make the most sense for it to be a new project (EWBIK-C#), or a fork of this project with some branches (branches/c#-double branches/c#-float branches/c#-unity-native-types) or some other variations?

Hmm. I can't really come up with any reason why users would want a jar and java source in some dusty directory their C# code never touches, so I think a new project would probably be the more sensible approach? Preferably with a link in the readme back to this project or something so people can check for (and maybe help with) feature lag between the two.

The double version seems to behave correctly, the float and Unity.Vector3/Unity.Quaternion versions behave the same and is captured in the above video where things just fall apart. If I move the target around, the skeleton updates and attempts to follow, and gives the impression it is generally working relatively well, but something somewhere is causing this constant issue which completely destroys the implementation. Hopefully, that is all clear.

I think I understand, yes. Though just to be clear, the Vector3/Quaternion version is sufficiently fast for your purposes, and the issue right now is primarily stability correct?

[Scraft]

I have seen this exact issue in the past where maths code in a C# script performs 10X (or more) worse than the same code written natively, put in a DLL and called from the C# script. There is a large amount of Garbage Collection being generated and cleared up too.

Ah, well that solves that mystery. Though, with regard to the Garbage Collection, this code's current but somewhat inconsistent philosophy is "whatever, that's what the garbage collector is for." So optimizations are possible.

Yes, and on my end, I'm hopeful I can find more optimizations. I will look more into the profiler side, I have just avoided diving too deep into any one area whilst things are in flux. Seeing the performance win from using Unity Native stuff makes me feel that is the direction I want to end up going in (and it makes sense anyway, better to use a single Vector/Quaternion/Maths lib than have a new version bundled).

That will naturally replace anything I have looked at so far, that said, if you can suggest anything for me to look at, from the above video, I am happy to take a look.

Nope! The video you just posted is perfect. I've seen that drift issue before and I believe I know how to fix it. The only thing I'm unsure of is if it will still need fixing in the new version. (The solution is to just orthonormalize each bone's local axes every time it gets touched by iterateCCDStrand(), and possibly forcibly translate each child back to the parent bone tip [and I can give you more detailed instructions if you'd prefer to try it ahead of time rather than just wait for the new version]).

That's a relief! I was worried you would have no idea and it would be a tricky problem to work out how to track down and resolve. If there is a straight forward way for me to test your idea I am happy to do so, equally I am happy to wait for your big code drop and see whether that resolves the issue or not. Although it took a while to get the code working the first time around in C#/Unity, I have spent long enough porting and reporting and fixing issues that I suspect I'll be able to bring over the latest version without too many issues (unless brand new complicated issues spring up, which can happen sometimes)!

Oh: a little question, in terms of this C#/Unity version, I am more than happy to put the source on GitHub, would it make the most sense for it to be a new project (EWBIK-C#), or a fork of this project with some branches (branches/c#-double branches/c#-float branches/c#-unity-native-types) or some other variations?

Hmm. I can't really come up with any reason why users would want a jar and java source in some dusty directory their C# code never touches, so I think a new project would probably be the more sensible approach? Preferably with a link in the readme back to this project or something so people can check for (and maybe help with) feature lag between the two.

Perfect, I'll wait until the new code drops, then get the port working, once things are looking good I'll get it into a GitHub project and link it back to this main project.

The double version seems to behave correctly, the float and Unity.Vector3/Unity.Quaternion versions behave the same and is captured in the above video where things just fall apart. If I move the target around, the skeleton updates and attempts to follow, and gives the impression it is generally working relatively well, but something somewhere is causing this constant issue which completely destroys the implementation. Hopefully, that is all clear.

I think I understand, yes. Though just to be clear, the Vector3/Quaternion version is sufficiently fast for your purposes, and the issue right now is primarily stability correct?

Well, it isn't fast enough as is, but as is it is doing 25X iterations per frame, so getting that down to 5X iterations is going to help a lot. Further to that, Unity has a 'Burst Compiler' now, which can compile subsets of C# down to natively level performing bytecode. It isn't an area I have looked into very much, but I think this will be a good excuse for me to look into it. Even without the Burst Compiler, I will look further into the profiler and see what can be done. It feels like it should be possible to get a decently performing project though. I need to be able to target 90 FPS for Virtual Reality, and I can't use the entire frame time calculating IK, however, my hope is it will possible to get this running at a decent enough rate. But yes, before anything can happen, I need to get it stable using floats. Once the stability is there, I will be able to commit and have a baseline to compare any optimizations against.

[EGjoni]

Oh, just to keep you in the loop I'm hoping to push the new version maybe tonight or tomorrow. But, I was thinking about your use case last night and I'm concerned you have a strange additional challenge that might require you to do something additionally clever on top of just the IK system. Specifically, while you know the positions and orientations of the head and hands, you probably don't have advanced knowledge of the physical proportions of the body they're attached to, and this can have significant impact on pose quality (most worryingly, if your user is shorter than you anticipate, the virtual armature's spine will basically crumple inward, because as far as the solver is aware, your user isn't short, they're just trying to retract their head into their own torso).

I think you might be able to alleviate some of that issue by having some callibration step where you ask the user to mimic some prespecified poses so that you can infer their proportions -- but do be aware that it is something you'll likely need to think about.

I'm adding a few minor features as I clean up which should hopefully make it easier to avoid unnatural configurations resulting from minor deviations, but the problem domain is interesting and I'm vaguely beginning to suspect that you might find that constraining your armature won't be the best approach, or might even be counterproductive for your purposes. What you might end up finding gives better results is first creating a couple dozen prespecified known-good poses corresponding to different end-effector positions. Then doing some procedure like

1. for each sensor, i.take the four poses whose effector's bound that sensor's position in an n-simplex (probably a tetrahedron). ii. find the barycentric coordinates of the sensor within that simplex. iii. pose the armature to the weighted (by corresponding barycentric coordinate) average of those bounding poses.
2. average the three poses from those sensors.
3. run the IK solver unconstrained beginning from THAT averaged pose.

I think I remember reading a paper that did something like that a couple of years ago, I'll see if I can find it.

[Scraft]

Oh, just to keep you in the loop I'm hoping to push the new version maybe tonight or tomorrow.

Amazing, thank you, I look forward to porting it over to C# / Unity and seeing if the floating point issue has been resolved.

But, I was thinking about your use case last night and I'm concerned you have a strange additional challenge that might require you to do something additionally clever on top of just the IK system. Specifically, while you know the positions and orientations of the head and hands, you probably don't have advanced knowledge of the physical proportions of the body they're attached to, and this can have significant impact on pose quality (most worryingly, if your user is shorter than you anticipate, the virtual armature's spine will basically crumple inward, because as far as the solver is aware, your user isn't short, they're just trying to retract their head into their own torso).

Yes, so at the moment, with a basic FABRIK solver that is in now, there are avatars floating around, clearly the avatars were not designed for my 6'5" stature! So the initial plan was to have a calibrate button, the user stands naturally, presses calibrate, and then the avatar is scaled so the head sensor position matches the avatars head position whilst keeping feet grounded. I had appreciated that the entire skeleton wouldn't be a perfect match, but it wasn't clear how much of a problem it would cause. It sounds like more of a problem than I hoped. As an initial pass I could also ask the user to stand with arms fully extended horizontally, to get arm length, and then to bend elbows 90°, to get the distance to the elbow. As we only have tracking data for head and hands, perhaps these measurements would do enough?

I think you might be able to alleviate some of that issue by having some callibration step where you ask the user to mimic some prespecified poses so that you can infer their proportions -- but do be aware that it is something you'll likely need to think about.

Let me know if you feel more than the above would be required.

I'm adding a few minor features as I clean up which should hopefully make it easier to avoid unnatural configurations resulting from minor deviations, but the problem domain is interesting and I'm vaguely beginning to suspect that you might find that constraining your armature won't be the best approach, or might even be counterproductive for your purposes. What you might end up finding gives better results is first creating a couple dozen prespecified known-good poses corresponding to different end-effector positions. Then doing some procedure like

1. for each sensor, i.take the four poses whose effector's bound that sensor's position in an n-simplex (probably a tetrahedron). ii. find the barycentric coordinates of the sensor within that simplex. iii. pose the armature to the weighted (by corresponding barycentric coordinate) average of those bounding poses.
2. average the three poses from those sensors.
3. run the IK solver unconstrained beginning from THAT averaged pose.

I think I remember reading a paper that did something like that a couple of years ago, I'll see if I can find it.

Certainly with basic FABRIK, without correct bone dimensions, and limited constraint support (some homemade constraint support has been added, a bit of a WIP with lots of limitations) various issues were coming up, and it felt like the issue was whilst the solution reached was possible to mimic, it simply wasn't natural. It felt like we needed to work out a hint system, sort of like if the hands are in front and close to the chest, elbows are more likely to point outwards (even tho you can point your elbows downwards and reach many of the same points). I wonder if your suggestion could be an improved way to resolve this as we would essentially be suggesting these hints via the initial pose we solve from. Any helper methods you can add to achieve the above, or samples, would be highly appreciated. IK is a domain with quite a lot of deep knowledge required, lots of solutions available to achieve a basic, constraintless, rotationless, solution, but when you have a real world case with constraints, desired poses to reach, etc there doesn't seem to be so much around. I was surprised to i. find this project and ii. for it to be so actively supported. I don't know for sure, but I suspect the C# / Unity port may get more eyes as it has become the defacto way most people get into games development now.

[EGjoni]

Let me know if you feel more than the above would be required.

You might also want arms stretched sideways while shrugging, arms stretched upwards while shrugging, arms stretched downward while shrugging, and arms stretched forward while shrugging. With these you should be able to construct a sphere from four points for each arm. The center of this sphere, + (arms stretched outwards while shrugging - arms stretched outwards as much as possible) should roughly = the location of shoulder, allowing you to infer shoulder width.

Alternatively, you could ask the user to just touch the controllers to their shoulders, but this has the disadvantage of not being as funny to everyone else in the room.

Certainly with basic FABRIK, without correct bone dimensions, and limited constraint support (some homemade constraint support has been added, a bit of a WIP with lots of limitations)

I suggest not bothering trying to get constraints working with basic FABRIK. Deadlocks and race conditions are basically inevitable, and it's kind of annoying that the author simply published as if this weren't the case.

it simply wasn't natural. It felt like we needed to work out a hint system, sort of like if the hands are in front and close to the chest, elbows are more likely to point outwards (even tho you can point your elbows downwards and reach many of the same points).

Yeah, you're likely going to end up encountering the same sorts of "unnaturalness" out of the box with this library. Constraints can prevent poses that amount to hyper-extension, but they have no sense of "preference" or even "mild discomfort." There is a bunch of inactive / unfinished code in the new version for "soft" kusudama constraints, which essentially work by establishing some pressure to keep bones away from the constraint edges (here's an inactive visualization [which you should probably mute before watching]). I realized after like a day of working on it that the way I needed it to behave to not to be a pain for artists was mathematically impossible. However, I think the way you would need it to behave (presuming your use case doesn't require that keyframes take precedence over IK solutions) basically amounts to the trivial case, and should help keep things more natural. I can leave the bulk of that code in and maybe add some explanation, but I can't make any guarantee as to its stability.

Any helper methods you can add to achieve the above, or samples, would be highly appreciated.

Yeah, I'll have to think through the best data structure for it. Something similar would be useful from an artist perspective, so I'll need something general enough to be usable across domains. I managed to find that paper I vaguely remembered reading, but it turns out to be a bit too "out there" for most purposes..

I don't know for sure, but I suspect the C# / Unity port may get more eyes as it has become the defacto way most people get into games development now.

On that note, you said you'd tried a bunch of different solutions. Since you're working with Unity I imagine FinalIK would have been one of them. The videos I'd seen of it a couple of years back looked pretty robust, did it turn out to be limiting in some unanticipated way?

[Scraft]

I've noted your thoughts on shoulders and grabbed a copy of the PDF linked to.

The main issue with FinalIK is the license, having code that has no restrictions is always helpful.

The video is certainly interesting, it will be nice if we can get the Unity version synced with the Java version (including getting the same sort of tools you are demonstrating in the video, for editing and visualising joints, constraints, etc). At the moment my thinking is if you make smaller updates (after the big drop) they may be able to be manually applied to the C# in the appropriate places. If any large changes are made, then it'll be likely instead I'd reconvert the project (which involves running some scripts I have written and then fixing up a bunch of issues manually that I have become aware of). Either way, it would be nice if there was some parity - one step at a time of course. Perhaps we can work out some basic unit tests, i.e. if you import skeleton s and then move the IK effectors to a series of specific rotations and positions, does each bone appears at the correct place/orientation (where correct is within a low epsilon like tolerance).

[EGjoni]

The constraint visualizations are basically entirely in glsl. So porting to Unity should be trivial. I'll upload those with the next piush.

Unit tests are a good idea. I'll put in some of the JSON code I've set up for saving / loading armatures into the abstract classes. (I assume Unity/C# have reasonably painless JSON parsers?)

On the topic of parity, I realized this is going to be very inefficient to debug unless there's a float based version in Java, so I'm untangling a few things to make it easy to swap between the two. This means it'll probably be another day.

P.S. I think I've figured out a really good and simple way to give the user very reliable and versatile control over poses / rules across a wide variety of armature types, body proportions, and use cases.

[Scraft]

This all sounds very promising - it was on my mind that having a float Java version would have a lot of benefits for parity - I did try and get the Java version running myself but failed (I have done very little Java, with the majority of it being within Android apps, not standalone desktop stuff). So if you are able to run a float version at your end, we could get near identical results which should massively help track down issues.

[EGjoni]

New version is up. I've added but have not yet tested support for floats. Though I suspect drift won't be an issue.

I will upload an example humanoid armature with hands and head targets tomorrow. And probably clean the code a bit more.

With regard to porting, "Basis.java" and "AbstractAxes.java" are likely the places to start. Let me know if there's anything weird or confusing, some of the structure is there to make the library extensible to other java frameworks, and can likely be safely simplified when porting.

[Scraft]

Fantastic, I hope to have time after work tonight to take a look at this and see if I can get it ported over and tested!