New in-house full body IK system

Frooxius commented 1 year ago

Is your feature request related to a problem? Please describe.

The current system for full body IK for avatars is a heavily modified port of FinalIK. As such, this system has a number of issues:

The IK doesn't behave well in a number of different ways. Many of which were tweaked and fixed, but many of which remain (scrunched neck/spine, hips not responding well, shoulders being too stiff) - and some that were introduced by the tweaks
The implementation doesn't mesh well with the way engine operates, due to being originally written for Unity and ported over, making it hard to work with the codebase and make further changes, add features or implement various optimizations
There are potential licensing issues in the future, if we re-license the engine to other parties (e.g. letting users export standalone builds of their worlds) or we open source parts of the engine

Describe the solution you'd like

After digging around the FinalIK-based code for a long time, I believe the best and overall easiest solution is to take full ownership and build our own in-house full body IK system from scratch, using the accumulated knowledge of IK systems, issues that users have with various avatars and full body tracking setups.

This way we solve all the outlined problems at once and get a good, clean codebase designed to integrate well into FrooxEngine to work with into the future.

There are number of design goals for this IK:

Increased effector count - head, hands, feet, knees, hips, chest, elbows and shoulders - being able to move shoulders with trackers is something that has been requested in the past
Graceful handling of compressing and stretching the spine - avoiding neck scrunch and pulling the head away - making sure the head position matches 1:1 is important (e.g. when receiving nose boops in full body!)
Rig retargetting - when using avatars that do not match human body proportions, we'll retarget the rig on computations, to ensure it behaves well - e.g. when the avatar has extremely short legs, we'll solve it with long legs and then shorten them back
Arbitrary orientation support - the solver won't assume that the player is standing upright, making sure that parts of the body do not get twisted when being upside down, lying on back with feet in the air and so on
Input poses are human sourced - the IK assumes that the poses for body parts come from human and thus are subject to natural human body constraints. As such it will not enforce those constraints on its end - e.g. it will not prevent how far you can move your head, because it assumes that you physically can't move your head to the point it would be unrealistic
Arbitrary limb count support - one arm, four arms, four legs. Design the system so it allows adding/removing limbs - this would only be for the standard places those limbs are in though - however we might also find that it's easier to just provide additional IK components to achieve this instead, so this might end up not being a part of the core IK, but an addition on top
Potential support for quadruped/taur and other setups - during design phase, I'll see how easy it would be to generalize the system to support wider variety of rigs to make those kinds of avatars easier. No promises here though, in case this would come at the cost of the IK behaving well for bipeds

There are some smaller ones, but these are the key points. I'll add more info as time goes when necessary too.

Describe alternatives you've considered

We considered just keeping FinalIK and making incremental improvements to it, but this would only compound the problems and it's increasingly harder to work with. To solve some of the problems we were facing, we'd have to redesign how FinalIK is architect-ed in the first place, essentially rewriting it with a custom system anyways.

Alternatively we could also look for other 3rd party systems, but generally we'd run into the same issue there - they do not integrate super well with FrooxEngine and its needs. The IK is one of the key systems for platform like Resonite and something that's used daily for people to express themselves and inhabit their avatars, so having an in-house system tailored for the needs of the platform and with codebase we fully own and control is going to be highly beneficial in the long run.

Additional Context

As this system will be developed, there will be a period of overlap, where both FinalIK and our custom ones exist. This will allow for comparisons and easy testing as the custom system is iterated on.

Once our IK achieves feature parity and behaves well enough for most avatars, we'll perform switch and make it the new default. Following which we'll start phasing out FinalIK - converting existing avatars to the new system and eventually removing FinalIK completely.

Additionally I expect as components and algorithms are written for this IK, we'll trickle some of them in as individual components too - e.g. trigonometric and FABRIK solvers, that can be used for any chains and/or to build custom systems, so there will be some nice goodies to play with along the way too.

Frooxius commented 1 year ago

Related thread with info, comparisons and feedback: https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/130

alexderpyfox commented 1 year ago

Will this also incl #15 which is hand grip finger curl limit? So you can set how far the fingerss can curl inward when gripping button is pressed this fixes so fingers don't come out the back of the hand on some avatars

alexderpyfox commented 1 year ago

Would a new rig need to be made for avatars or will it be backwards compatible with the current rig naming etc?

ModernBalloonie commented 1 year ago

Related thread with info, comparisons and feedback: #130

Thank you!

I'm going to do more comparisons in here from now on when I have the time to do so!

TehTurk commented 1 year ago

With this issue, would it be also ok to generalize and expose IK components as well? This would cut back on requirements for the issue somewhat, and also allow folks to make their own custom iks a bit easier (Taurs, Multi-peds, other systems outside of arms)

Twist bone support is also something I realize shouldn't be missed as well.

Frooxius commented 1 year ago

Will this also incl #15 which is hand grip finger curl limit? So you can set how far the fingerss can curl inward when gripping button is pressed this fixes so fingers don't come out the back of the hand on some avatars

No, that's not part of this. Fingers are a separate system.

ModernBalloonie commented 1 year ago

With this issue, would it be also ok to generalize and expose IK components as well? This would cut back on requirements for the issue somewhat, and also allow folks to make their own custom iks a bit easier (Taurs, Multi-peds, other systems outside of arms)

Twist bone support is also something I realize shouldn't be missed as well.

Adding onto this, I do wonder if manually assigning rigs could be a thing too because for example: unity has this:

So while you should be using proper bone naming, if you don't you won't be completely screwed and you could reinitialize the IK by setting up some of the bones manually. Sometimes avatars for other platforms have some issues in Resonite due to the person naming bones improperly, and instead of having to reimport the whole thing it would definitely be nice to be able to do that.

Frooxius commented 1 year ago

With this issue, would it be also ok to generalize and expose IK components as well? This would cut back on requirements for the issue somewhat, and also allow folks to make their own custom iks a bit easier (Taurs, Multi-peds, other systems outside of arms)

Twist bone support is also something I realize shouldn't be missed as well.

This has been touched on in the initial post. The goal is to generalize things where possible, but it will be avoided if it comes at the cost of performance, the IK behaving well for humanoids (which is the primary goal).

TehTurk commented 1 year ago

Ah okay cool! Sorry missed that bit about the generalized components!

Frooxius commented 1 year ago

I've created a discussion for this, to keep things better organized. Please direct feedback, thoughts, wishlights and other stuff here: https://github.com/Yellow-Dog-Man/Resonite-Issues/discussions/616

Dusty-Sprinkles commented 5 months ago

Wondering cause I wanted to make a new issue for it but I don't know if a better solution is in the plans for the new IK, but would avatar setup/post setup still behave about the same? I wanted to create a suggestion/feature request to add some kind of button that lets you preview an avatar's viewpoint during the setup process to make placing the eyes a bit easier

shiftyscales commented 5 months ago

The avatar creator itself is due for an overhaul/rewrite, but that is a separate issue/work item from the IK itself, @Dusty-Sprinkles.

I talk about it a bit here: https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/20#issuecomment-2078294380

Yellow-Dog-Man / Resonite-Issues