AI system to extrapolate full body motion based on a limited number of trackers, trained on mocap and/or full-body tracking?

[TiagoTiago]

Inspiration: https://www.youtube.com/watch?v=X5Z7ZJ39zAA (of particular interest, the mocap reconstruction demonstration starting at around 2:41 )

That video is from many years ago, I imagine the technology has advanced a lot since then.

The way I picture it, it would be something like, you would load up one or more "styles" (for example, gendered walking style, a dance moves style, a "pretending to be a monster" style, "kitty" style etc), and then based on whatever number of trackers you have, the relative position of the trackers to each other and things like vertical position relative to user height, roomscale movement of the head, plus recent previous positions of everything; the system would figure out the most likely motion for the untracked parts of the skeleton. And ideally, people would be able to train new styles using standard mocap formats, and possibly also by capturing full-body tracking with VR gear (at least your own, but being able to record people with true full-body tracking you meet automatically to improve a general motion style would also probably be a good option to have).

[TiagoTiago]

I think this is the paper mentioned in the vid: https://grail.cs.washington.edu/projects/styleik/

[Frooxius]

Is there any existing library/solution for this? I'm not sure if this would actually work all that well with VR for immersion over existing methods or simply just using full body trackers.

This is not something we'd likely implement from scratch, given the amount of work involved, so it would be very low priority.

[guillefix]

There is one company I know which offers what you want as a service https://deepmotion.com/virtual-reality-tracking I don't know how expensive their licensed software is though.

---

I am actually working on a general AI/ML plugin for Neos with a current focus on mocap data (because that's the data most unique to VR!). I'm taking a bit of a break on the project on August, but will begin working on it after that. My focus is on making autonomous agents, but perhaps it wouldn't be too hard to extend it to extend to generating motion conditioned on points controlled by the user.

I was originally using imitation learning (GAIL) but that turns out to be too brittle of a technique, so I'm gonna try the approach in https://github.com/sebastianstarke/AI4Animation and specially this newer paper by Ubisoft https://twitter.com/anorangeduck/status/1289310169797517313?s=21 (as the first one is rather complex).

I've seen a paper that does the same thing as your video, but using neural networks. I like the simplicity of the approach where they only learn a representation of poses (rather than movement). Hmm that may be a simple initial thing to try with my plugin.

[TiagoTiago]

I imagine that adding recent previous positions, basically a very short-term history, to the training/inputs would likely allow for a pose based system to behave sorta like it's actually thinking in terms of motions since each time a new position is calculated, it is taking in consideration what the bones were doing moments before; and if you include in that, the positions the system itself generated, it might even be capable of generating things like idle motions and stuff even when the user inputs remain static.

[Frooxius]

@guillefix That one looks interesting. But yeah if they don't mention price it'll probably be very high. It's something we might explore in the future, but not for a while.

@TiagoTiago I'm sorry, but I don't really have any plans on implementing any such algorithm myself at this point, our priorities and focus lie elsewhere for the immediate future.