Suggestions: Limiting Pose, Adding Bake

[KaijuKoder]

Been testing BlendARcap alongside other Mocap systems and here is what I found:

TLDR : Suggest the ability to bake animation so the user can clean it up, and restricting body motion so Face/Hands can be used just with upper Pose(body). You can go far with a decent taking head. Right now face alone doesn't animate the neck so without the body it looks weird.

• BlendARmocap Face and Hands detection, from either webcam or video, works brilliantly. No crashes. No Bugs. It just works, and it's fast. Very nice.

• BlendARmocap Pose(body) detection kind of works, but has some big issues which limits is usefulness.

Jitters: The result is really jittery. Even limbs and body parts known to be still jitter and jump around. However the basic pose is there. It's not far off.

The jitteriness can perhaps be smoothed by averaging points over time. That could be done either by BlendARmocap, or afterwards by letting the user simplify the curves of a baked animation. It'd be more satisfying to fix in straight out of BlendARmocap, but all Mocap tools I looked at have this problem to some degree, and I think you will always need do post-processing anyway.

Limiting Body Parts: Another possibility for reducing jitter is constraining body parts. If we know the human is sitting, then we don't need to move their hips. If their legs are still, you can ignore all animation there. Same with feet. Many mocap systems allow you to freeze the feet.

Because BlendARmocaps Face and Hands detection is so good, limiting the body movement to just the head, neck, shoulders and arms would make BlendARmocap a very useful tool, because a lot of shots are just "talking heads" anyway. Your interface is nice and clean. It's already quick and very easy to use. You've processed it faster than the time it takes to transfer the files to a cloud mocap system!

Weird Posture: There are also orientation issues with the body. If you're shooting at a (say 45 degrees) angle, which you really need to do for coverage when you've only got a single camera, it gives you a weird body posture, like you are sitting diagonally. I tested Plask.ai and Rokoko Video and they had a similar problems.

Using multiple cameras seems the way to go here, and FreeMoCap which does that works really well, but at the moment FreeMocap appears to have lots of problems with dependency issues, python bugs and GUI problems. At this time, it's not really useable.

Nevertheless using multiple cameras for 3D seems a very good solution. Have you considered this? The ability to hook up to FreeMoCap's data would be useful, but only if FreeMocap was working.

Reference Pose: Am wondering if beginning a video with a reference A or T pose by the human could help the software detect the position they move into (like sitting). Cloth simulation systems use this technique.

FK vs IK: Did notice in the transfer seems to be using IK on the arms? Would suggest FK is better because otherwise you get the elbow bending through the body.

Need Bake/Export: Something that really is needed is the ability to bake the animation into conventional keyframes so they can be cleaned up. Plask.ai and Rokoko Video have similar problems to those I described for BlendARmocap, but their ability to export a FBX of the baked animation you can clean up yourself is a big help.

Retargeting: Looked at much retargeting software. Todor reports Rokoko's addon is broken as of Blender 4.3, and their advice to animate the metarig is bogus. I tried Paolo Acampora's Expy but as yet haven't been able to get a good result out of it. Much retargeting software assumes the rigs are the same sizes, so even minor differences cause really bad results.

Video conditions: Found a bright room, clothing with patterns on it, and if your software allows it - multiple cameras - give best results.

Suits: I was considering buying a suit to circumvent this, but seems we're not at the golden age of Mocap just yet:

Made for Patreon: Rokoko Vs Xsens https://www.youtube.com/watch?v=LVtmgwzve8Q

I wasted $4000 on a Rokoko Mocap Suit https://www.youtube.com/watch?v=LKkOT1z54XM

So a multi-cam markerless video solution may end up being better anyway. Rokoko Video and Plask.ai do a decent job, but there are transfer delays and the web interfaces are clunky. If you can do it quickly on your own PC which it seems BlendARmocap can, then all the better.

[cgtinker]

Thanks Kaju!

Many of those things I've addressed - I really need to make some tutorials soon. Currently the new version can be downloaded from the main branch - it's not available as release yet. My suggestions are based on the upcoming release (available by downloading the main branch as zip).

1. To reduce Jitter jitter by averaging: select the objects yielding the animation, navigate to the Object Constraint Properties tab > BlendArMocap > Tools > Smooth Animation. This may be done before baking.
2. Limiting Body Parts can be done by either just deleting the objects you don't want to transfer before transferring (e.g. the _cgt_hipcenter empty or navigate to the Object Constraint Properties tab > BlendArMocap > Drivers and select None.
3. Reference Pose doesn't matter to much when using IK, when using FK I'd stick to T-Pose. Currently there is no FK preset file (I may add this soon) but you can use this setup helper script - just add the generated rigify rig name, set _useik to False and _usefk to True - then run it.
4. To improve posture consider to use multi cam tracking. Checkout freemocap - you can export the recorded session data from freemocap and import it using BlendArMocaps Freemocap Import.
5. To bake animations you can either use Blenders backing system, if you want to transfer the animation to the rigify metarig (which may be used as gamerig) checkout my add-on rigify_gamerig_extension

Never tried Tracking suits, sorry for your loss - I'm surprised the results don't live up to their expectations. With the collab between freemocap and blendarmocap I think mocapstudios can be setup at home in the near future. :)

[KaijuKoder]

Thanks @cgtinker. I was able to get baking working. I'm currently experimenting with retargeting and cleaning up the animation. It's looking promising. I'll let you know what I find.

Unfortunately the metarig isn't much use for doing Pixar-like video. Really need to transfer to a decent control rig where it can be further animated. Transferring the FK is relatively easy, but there are some issues with IK controls when the mocap and target rig differ in size, which will almost always be the case. Even small differences really mess things up.

I'll submit bug reports to Freemocap.

Yes, if you pay that much for a tracking suit, you'd expect it be flawless. However the human brain is very good at recognizing poses with only binocular vision and if you're ever manually keyframed an animation, you've done it with monocular vision. Using vision AI seems a superior solution.

[cgtinker]

Yeah baking animations to the metarig is mainly intended for the export to game engines.

You can also bake to the upgraded rig without issues. Just go in pose mode pose > animation > bake action - make sure to select visual keying, besides set it up to your likings. Then stash the animation to the nla editor.

[KaijuKoder]

1. Been stuck on retargeting. Found many, many retargeting addons for Blender. Some get close, but none get close enough to be useful. AFAICT none do proper retargeting. They all only use bone constraints (from copy transform, copy rotation, copy location, armature) to copy the source bone to the target bone. The locations present problems unless your source and target rig are identical, you get really weird mesh deformations. All the Youtube tutorials which tell you how fast and easy it is use identical rigs!

Think the current retargeting tools for Blender need to be considered first generation. People ask why bigger studios aren't using Blender. This is one reason.

I did discover of those bone constraints, discarding the location ones and using only the rotations works fairly well. You don't need a lot of bones to make it work well either: just head, neck, shoulders, arms and legs. Not perfect, but close enough to give you a good start for the scene. Unfortunately at the moment BlendARmocap uses IK for the limbs which prevents this. When I have a chance I'll have a look at the configuration file you suggested to support FK.

I did try doing FK->IK for each frame from a python script. Wasn't too bad, but has pole issues and won't match what the limbs really did. FK would be better.

2. Looked at the proper Rigify rig (not the metarig which IMO isn't a real rig even if people are using it as one), and found it's easy to bind to that. The legs and arms, head and neck match one for one. Using the Mixamo naming convention Spine 2 binds to the rigify shoulders. Unless you specifically want a stick figure for a game, there is no reason not to use the proper Rigify Rig. It gives you much extra control over the finished product.

3. Did find something strange in my tests with BlendARmocap. The head is rotated 90 degrees left more than it should be. That's on a standard rig, so not a retargeting issue. Not sure if it just my test case. Gone when reran.

4. More on the configuration problems. I found when there is a missing prerequisite if I go to C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\bin\python.exe and install it manually via .\python -m pip install PACKAGENAME, instead of putting the package in C:\Program Files\Blender Foundation\Blender 3.4\3.4\python\lib\site-packages, Python puts it in C:\Users\guest\AppData\Roaming\Python\Python310\site-packages instead. So I have to explicitly add that with PYTHONPATHS to my system environment variables. If I remove that, it doesn't work. I have it working now, but it's weird.

Update: Continuing to test. Apart from the FK/IK issue (I'm looking at the configuration file now), and some strange behavior on certain bones (Hips seems to bend the wrong way on the Rigify rig when sitting), it's doing very well. Threw a fairly difficult looking dance clip at it and it did really well.

[KaijuKoder]

Been doing more testing. The results are very good. Apart from the IK issue I describe below, it holds its own to Rokoko Video and Plask.ai which is an accomplishment. Considering BlendARmocap runs locally on a PC without banks of cloud servers and transfer delays, that is a real achievement.

The IK issue is with hand placement. I find it on any rig. TLDR: The IK hands appear in the wrong place.

To show this attaching a video test I ran comparing the same reference video as processed by Rokoko Video (left), and BlendARmocap (three right) shown on (left to right) on standard rigify rigs: unedited rig (no mesh), a stylized little guy (small stature but with slightly longer than usual arms and legs), and a big guy (Thanos-like, "Male basemesh 04" by Teo is licensed under Creative Commons Attribution), processed only in BlendARmocap Pose mode. For clarification, I added the FK keys to the IK bones. Each of these was transferred separately and baked so you can see them running alongside.

Note the right hand. It begins flat on the character's right thigh. Rokoko Video gets it right, but on all the BlendARmocap rigs it is off. On the standard rig, it's way back, near the hips. On the little guy, it's way forward and past his knee. On the big guy, it's crossed over into the front of the left thigh.

https://user-images.githubusercontent.com/107446154/217146939-610f672d-c943-4346-8326-5a29e44868f8.mp4

I did write a script to compute the FK keyframes from the IK positions, but because the IK positions are so far out, it didn't help, though it does let you better see what is going on:

 context.scene.tool_settings.use_keyframe_insert_auto = True

 rigify_rig_id = context.active_object.data['rig_id']

 rigify_snap_fn = getattr(bpy.ops.pose, f'rigify_generic_snap_{rigify_rig_id}')
 if not rigify_snap_fn:
     raise ValueError(f'Error: Rig operator missing. Have you run its rigify script {context.active_object.data.name}_ui.py?')

for frame_nr in range(context.scene.frame_start, context.scene.frame_end+1):
  context.scene.frame_current = frame_nr

  print(f'Snap FK->IK for frame {frame_nr}')

  result = rigify_snap_fn(input_bones="[\"upper_arm_ik.L\", \"MCH-forearm_ik.L\", \"MCH-upper_arm_ik_target.L\"]", output_bones="[\"upper_arm_fk.L\", \"forearm_fk.L\", \"hand_fk.L\"]", ctrl_bones="[\"upper_arm_ik.L\", \"upper_arm_ik_target.L\", \"hand_ik.L\"]")
  result = rigify_snap_fn(input_bones="[\"upper_arm_ik.R\", \"MCH-forearm_ik.R\", \"MCH-upper_arm_ik_target.R\"]", output_bones="[\"upper_arm_fk.R\", \"forearm_fk.R\", \"hand_fk.R\"]", ctrl_bones="[\"upper_arm_ik.R\", \"upper_arm_ik_target.R\", \"hand_ik.R\"]")
  result = rigify_snap_fn(input_bones="[\"thigh_ik.R\", \"MCH-shin_ik.R\", \"MCH-thigh_ik_target.R\", \"toe_ik.R\"]", output_bones="[\"thigh_fk.R\", \"shin_fk.R\", \"foot_fk.R\", \"toe_fk.R\"]", ctrl_bones="[\"thigh_ik.R\", \"thigh_ik_target.R\", \"foot_ik.R\", \"toe_ik.R\", \"foot_heel_ik.R\", \"foot_spin_ik.R\"]")
  result = rigify_snap_fn(input_bones="[\"thigh_ik.L\", \"MCH-shin_ik.L\", \"MCH-thigh_ik_target.L\", \"toe_ik.L\"]", output_bones="[\"thigh_fk.L\", \"shin_fk.L\", \"foot_fk.L\", \"toe_fk.L\"]", ctrl_bones="[\"thigh_ik.L\", \"thigh_ik_target.L\", \"foot_ik.L\", \"toe_ik.L\", \"foot_heel_ik.L\", \"foot_spin_ik.L\"]")

I guess the IK hand placement is off during the transfer. Not sure if it is better to find out why it is off, or just to convert the configuration file to do FK. Each approach has its merits. Easiest thing to do is probably FK, but a better (harder) solution would be to combine both: Try FK first and if the hand reaches the IK target, then use the FK. If it doesn't, then relax the FK and use IK - assuming the IK target is correctly placed. Ideally long term you'd test for mesh collisions but simple and working beats ambitious and not. I guess try converting the configuration file to FK first and see how much it improves.

All told though: It's looking very good. Every Mocap solution I looked at needs clean up before you can use it, although obvious the lesser is better.

[KaijuKoder]

@cgtinker, TLDR: Tested FK and it seems good to go with a minor mod (swap left and right)

Tried using the tf_pose_mapping_helper.py script as you suggested, but didn't seem to do anything. So added bone constraints manually after transfer step:

upper_arm_fk.R -> COPY_ROTATION cgt_left_shoulder with defaults forearm_fk.R -> COPY_ROTATION cgt_left_elbow with defaults thigh_fk.R -> COPY_ROTATION cgt_left_hip with defaults shin_fk.R -> COPY ROTATION cgt_left_knee with defaults didn't try foot_fk.R, toe_fk.R, wrist_fk.R, but assuming the same and vice versa on the other side

Note I needed to use the left cgts on the right side, and the right cgts on the left side! Note the same video was what I used with Rokoko video and BlendARmocap IK and it wasn't reversed. Yet BlendARmocap FK was. Don't know why, but doing the above fixed it.

I looked closely at the posturing errors:

1. The IK hand position. Interestingly I found it the IK position exactly matched the FK position! I guess that upstream that IK hand position is calculated entirely from the FK bones, instead of being calculated by its position relative to other landmarks. If that's the case, it's perhaps better to use FK rather than IK anyway, since its derived from the same source, but without the benefit of elbow/knee data. TLDR: Just use FK anyway?

2. Strange poses. When I looked closely at these - like this video, I realized it was getting the limb positions correct! The problem was the differences in the rig size. Even a few centimeters out can introduce huge errors. I think it's to okay to leave the cleanup of that to the user, since none of the Blender retargeting solutions I looked at properly handles different bone lengths, etc anyway.

https://user-images.githubusercontent.com/107446154/217431113-5a858e4d-d642-4a8f-8f30-ec72542ec72e.mp4

Summary: I think adding FK in its current form would be fine, and perhaps even better than IK.

I'll test hand and face transfer next.

[cgtinker]

@KaijuKoder Really nice input, thanks a lot!

Let's discuss the mapping / setup helper By default, I mirror everything (L -> R), I follow mediapipes naming conventions and by now, mirroring back everything would just create unnecessary work.

I think I forgot to mention that the tf_pose_mapping_helper.py doesn't transfer itself - it just sets up a transfer config. You got to press the BlendArMocap > Tools > Transfer Selection button in the object constraint properties to transfer such a preset config. You manually did the fk transfer using constraints, which accomplishes the same thing but cannot be saved as preset config and therefore cannot be reused.

Example If you select an empty like the cgt_hip_center and navigate to the constraint properties, you'll find a lots of mapping properties in the BlendArMocap Tab. You can easily offset the angles. Besides, currently there's no option to offset FK angles, maybe I'll introduce it in the future.

The concept here is to generate a driver in blender based on the properties and using a constraint to copy values from the driver. Constraints from the empty get applied to the bone and target the driver. This introduces way more flexibility and adaptability than just using constraints. You'll find more details in the README.md.

This way you can also target special rigs and setup your own custom mapping presets. To store and reuse it use the 3D View > BlendArMocap > Transfer > Save Config button.

Besides, I'll add an ik & fk transfer option. 👍

[cgtinker]

About the calculations: IK: The tracking results get remapped based on the rigs bone lengths, therefore the mapping may only be accurate for the rig which has been transferred to initially.

FK: the rotations are basically shoulder > elbow > wrist expecting a T-Pose rig. This will most likely lead to issues with A-Pose rigs.

[KaijuKoder]

@KaijuKoder Really nice input, thanks a lot!

Thank you. I think BlendARmocap will turn out to be a very useful tool for the Blender animation community. Perception is Blender has been lacking in animation tools, which makes it a less productive environment to work in: You can animate in Blender, sure, but at a plodding pace. Animation studios have been accused of not switching to Blender because of snobbery and a lack of imagination, but I think productivity is a big factor. IME if you offer a business a tool that lets them do something better, faster, cheaper than their existing tools, they will change quickly.

BlendARmocap has the potential to change things in favor of Blender.

Workflow I'm experimenting with: Use BlendARmocap and proceed to Transfer, Scrub constrained animation, choose keyframes, remove constraints, lets splines inbetween. Seems to work better than baking everything and cleaning that up, because retargeting really bogs you down, and while simplifying fcurves works, result is ok though not great which requires even more cleanup. Cleanup is as plodding as traditional animation.

Let's discuss the mapping / setup helper

Thanks for the explanation of tf_pose_mapping_helper.py. I'll try that next test.

I did find adding the FK constraints even manually got very good results. Also FK seemed to align the feet better, since the IK feet weren't picking up the rotation - See PNG - although perhaps this was a glitch for this particular test. I'll let you know if it persists.

Thanks for the explanation of the internals of BlendARmocap. I had tried reading it but couldn't follow.

IK: The tracking results get remapped based on the rigs bone lengths, therefore the mapping may only be accurate for the rig which has been transferred to initially.

Yes, Retargeting manually, while acceptable, does take a lot of time, and changing one thing has a ripple effect which requires changing many other things. So it does eat away at the advantages. I'm thinking there are easier ways to improve this without having to resort to detecting mesh collisions.

FK: the rotations are basically shoulder > elbow > wrist expecting a T-Pose rig. This will most likely lead to issues with A-Pose rigs.

Which is interesting because while MoCap systems seem to love T-Pose, these days everyone creates models in A-Pose - to minimize shoulder distortion, and because T-Pose is so rarely done IRL.

I'll do some more pose tests, and then move onto head / hands tests - which I expect will be much easier.

Update: Doing some more testing, and it seems the IK feet and Chest (shown in light blue) are always forwards. In this screenshot, the other bones including the torso are rotated correctly (sitting with body at origin and feet along +X), but not those:

[KaijuKoder]

More experimenting: This time with using FreeMocap with BlendARmocap. Found:

• FreeMocap alpha version is unstable and hard to use (no feedback) They recommend prealpha for stability, but was able to get it working though ran into the same bugs as prealpha:

• FreeMocap prealpha had stopped working due to changes in Mediapipe and dependency issues. Was able to get it working again by manually downloading certain Mediapipe assets which Google had relocated for the latest release, and playing around with Python dependencies: Uninstalled and reinstalled kiwisolver, matplotlib. Was able to remove PYTHONPATH. Will post on their Github.

That done, was able to open the FreeMocap session into . Kind of worked, but found some bugs:

• The axes seem wrong. The video was of a human standing, but the rig has been squashed as if they are lying down, but with certain bones (e.g. the feet) oriented as if they are standing. Note the BlendARmocap face control points are looking down.
• Also note those IK feet, and chest have the same problem as reported last time: They are facing forward. In this case (and maybe the prior one) the head had the same problem. The torso has the same incorrect orientation, though in the prior one it was fine. If I disabled the constraints on the problematic bones, it looked much better, though of course I lost a lot of the motion!

From above looking down

From below looking up. Note the squashed rig, and face control points looking down.

• Regarding the IK feet always being flat and facing forward, might be same issue with hands. Noticed them at a strange angle, but the FK hands seemed okay. Not sure. Will retest.

That done though, the result was promising. The points did appear to be in the right position in 3D space, and were much more stable.

[cgtinker]

Workflow I'm experimenting with: Use BlendARmocap and proceed to Transfer, Scrub constrained animation, choose keyframes, remove constraints, lets splines inbetween. Seems to work better than baking everything and cleaning that up, because retargeting really bogs you down, and while simplifying fcurves works, result is ok though not great which requires even more cleanup. Cleanup is as plodding as traditional animation.

I like the workflow you are going for, I've had something like that in mind swell.

I did find adding the FK constraints even manually got very good results. Also FK seemed to align the feet better, since the IK feet weren't picking up the rotation - See PNG - although perhaps this was a glitch for this particular test. I'll let you know if it persists.

I'll check the ik foot and ik hand rotations before the release, maybe something is stale, not sure about it. Setting up fk constraints manually should deliver the same results as using the setup script at the moment. Major benefit is that you can safe the setup and reuse it.

Yes, Retargeting manually, while acceptable, does take a lot of time, and changing one thing has a ripple effect which requires changing many other things. So it does eat away at the advantages. I'm thinking there are easier ways to improve this without having to resort to detecting mesh collisions.

Saving configurations really can help you there. When working with the setting in the object constraint properties all along, you can safe your work. To do so, navigate in to the 3D-View > Sidepanel > BlendAr > Transfer and chose safe config. When targeting another rig, select the config of your choice, consider to load it or just press transfer.

Which is interesting because while MoCap systems seem to love T-Pose, these days everyone creates models in A-Pose - to minimize shoulder distortion, and because T-Pose is so rarely done IRL.

True, I think the easiest way to fix this is to offset the rotation. While this should work with the existing mapping tools, I can imagine to calculate the rotation differences in the future and just multiply them apart. For now, it may be fixed using the remapping options.

Update: Doing some more testing, and it seems the IK feet and Chest (shown in light blue) are always forwards. In this screenshot, the other bones including the torso are rotated correctly (sitting with body at origin and feet along +X), but not those:

True I think currently everything is accepting the rig to face forward. This shouldn't be an issue with fk, however, I use similar concepts for the facial animation. Gotta give that some thought

[cgtinker]

• FreeMocap alpha version is unstable and hard to use (no feedback) They recommend prealpha for stability, but was able to get it working though ran into the same bugs as prealpha:
• FreeMocap prealpha had stopped working due to changes in Mediapipe and dependency issues. Was able to get it working again by manually downloading certain Mediapipe assets which Google had relocated for the latest release, and playing around with Python dependencies: Uninstalled and reinstalled kiwisolver, matplotlib. Was able to remove PYTHONPATH. Will post on their Github.

Thanks for doing so <3. Just seen their requirements, they should really switch their mediapipe version - the version in use has bad packaging which leads to bugs.

That done, was able to open the FreeMocap session into . Kind of worked, but found some bugs That really seems like the axes are wrong. Bummer, we made some tests and it seemed fine. Got to revisit that.

That done though, the result was promising. The points did appear to be in the right position in 3D space, and were much more stable.

Glad it worked in the end. I'll check why the axes are screwed.

[KaijuKoder]

Thanks @cgtinker. Please advise when you've finished the latest update and I'll test it. I've had to fall back on manual animation for now, but it's slow going.

Saving configurations really can help you there.

Is that after adjusting the object constraints on the drivers? I haven't tried those yet, because I'm not sure what to change there e.g. to accommodate a character with narrow shoulders or longer/shorter limbs than the original. The inability to retarget is why I fell back to manual animation. It's so much effort to clean up a badly targeted rig, faster to do it from scratch.

True I think currently everything is accepting the rig to face forward. This shouldn't be an issue with fk, however, I use similar concepts for the facial animation. Gotta give that some thought

I was filming at an angle because there are some postures better caught side-on than front-on.

they should really switch their mediapipe version - the version in use has bad packaging which leads to bugs.

I've suggested they tell people how to get the prealpha working, since the alpha can't run to completion (for me anyway). The good news is with the tweak posted, the prealpha can work: https://github.com/freemocap/freemocap/issues/302#issuecomment-1441177245

BTW I've had no more dependency problems since forever now. It seems the problem I had before was caused by them being installed in the wrong site-packages directory (Python has so many), and perhaps the PYTHONPATH variable.

[KaijuKoder]

TL;DR

• Allow much more smoothing, to zero out the jitter. A jittery rigs is practically unworkable.
• Provide foot plant and sitting checkboxes.
• Fix always-facing forward problem.

Been back doing some more experimenting. This time I tried a video where the character begins sitting, looks around, leans forward, etc.

I did it in BlendARmocap and compared it to Rokoko Video. Differences:

In BlendARmocap the bones jittered a lot. Increasing confidence helped, but only by so much. Wasn't able to use the workflow suggested in an earlier post, because the bones jittered around so much, it wasn't viable even to sample major keyframes, because the jitter made even similar postures (in the video) appear very different (in the rig). Rokoko seem to handle this with a very nice smoothing algorithm.

A workaround for BlendARmocap may be to export the enter animation, jitters and all, smooth those fcurves automatically, then go through and delete all non-major keyframes. But it'd be easier if the rig smoothed initially, saving the additional step. I'd even suggest smoothing aggressively, because you can always add extra movement to an over-smoothed rig, where as you can't do anything with a jittery rig (until you remove the jitter).

In the video, the feet were planted, and Rokoko kept them that way (their suits seem to have a problem with feet, and their software seem to put a lot of effort into mitigating that). In BlendARmocap's feet swung quite a lot, which made it hard to capture major keyframes. You did mention not transferring the leg motion, which I get conceptually, but I'm not sure how to do it in practice. Because sitting down is such a common thing, would help usability if there was a checkbox (same for standing with the feet planted). Would save the user a lot of time.

Note BlendARmocap still has the problem with the shoulders not rotating (i.e. staying pointing forward, even as the character rotates the rest of their body). Rokoko handled them properly. I did try videoing front-on to minimize the problem from this, but while MediaPipe makes a noble attempt, you lose too much side information.

[cgtinker]

allow much more smoothing, to zero out the jitter. A jittery rigs is practically unworkable.

Try the current version from the main - basically added a smoothing option while running movie detection. A Key Step of 1 is no smoothing, the higher the keystep, the more aggressive the smoothing (b.e. a keystep of 4 is already a lot of smoothing).

Fix always-facing forward problem.

There have been some driver / constraint issues with feet, shoulders, knees and elbows. Should be fixed now.

Provide foot plant and sitting checkboxes.

Well, as a transfer to other rigs is planned and the setup could be used differently I don't know about this. To not transfer the motion just disable the constraints on the legs or delete empties which are relevant for the legs before transferring (like deleting knees, foot index, ankle, hips)

[KaijuKoder]

Thanks @cgtinker. I'll try the new version.

Try the current version from the main - basically added a smoothing option while running movie detection. A Key Step of 1 is no smoothing, the higher the keystep, the more aggressive the smoothing (b.e. a keystep of 4 is already a lot of smoothing).

Thanks for explaining. One problem I have with Blender is tooltips often tell you what some numeric value does, but not in which direction it does it! :)

There have been some driver / constraint issues with feet, shoulders, knees and elbows. Should be fixed now.

Nice!

Well, as a transfer to other rigs is planned and the setup could be used differently I don't know about this. To not transfer the motion just disable the constraints on the legs or delete empties which are relevant for the legs before transferring (like deleting knees, foot index, ankle, hips)

Consider those long-term suggestions to make the software easier to use. Not as important as getting it working.

[KaijuKoder]

Hey, @cgtinker. Did some testing with a new video. I need to do some more, but quickly: yes, the shoulder seems better, though I notice the head and neck have no constraints now (I think they previously did?), so don't move independently of the torso which looks weird.

Experimenting with the drivers. I suspect I need to understand and tweak those to get a better rig match. The problem I found is if you bake to Blender with a bad offset in your rotations/locations, it's really too late to clean that up in your keyframes (because there is no way to say to Blender e.g. "rotate every torso keyframe forward by 30 degrees"). Secret is I think that you really need to have a good rig match before you bake.

Update: Realized the head is only included now when you transfer FACE, but really should be done for POSE too. Though got to say FACE by itself just did a really nice job.

[KaijuKoder]

Hey, @cgtinker, Back after a 2 month break doing other shots so just touching base.

1. Did try the live version but got install errors complaining it couldn't load google.protobuf. So tried the /releases version and that installed/worked fine.
2. Found after a 2 month break, I forgot how to get BlendARmocap working. Found myself trying to remember how to create and merge FACE and POSE/HOLLISTIC from the same video, and remove the constraints. Longer term for usability, would be nice if the workflow was smoother because even stuff like disabling the legs (which swing wildly even if they're not in the frame) breaks my brain. The face animation works well, but the POSE/HOLLISTIC still tends to flail so much due to retargeting that it's faster to do the shot manually (which is still slow).
3. Will do some more experimenting over the next week and report back.
4. Question: What is FACE_DATA - compared to FACE? Did notice some drivers make spots on the face move (which is good) but others don't.

[cgtinker]

Hey, @cgtinker, Back after a 2 month break doing other shots so just touching base.

1. Did try the live version but got install errors complaining it couldn't load google.protobuf. So tried the /releases version and that installed/worked fine.

What's the "live" version? The version from the main branch? Can you let me know the error message?

2. Found after a 2 month break, I forgot how to get BlendARmocap working. Found myself trying to remember how to create and merge FACE and POSE/HOLLISTIC from the same video, and remove the constraints. Longer term for usability, would be nice if the workflow was smoother because even stuff like disabling the legs (which swing wildly even if they're not in the frame) breaks my brain. The face animation works well, but the POSE/HOLLISTIC still tends to flail so much due to retargeting that it's faster to do the shot manually (which is still slow).

Merge Face/Pose/Holistic? I thought that's what holistic is for, I think I don't fully understand. I think the abstraction that's hard to grasp is that the data is just on empties, has something in-between and then gets copied by constraints. There are two easy ways of disabling:

1. remove the constraints from the rig (just go to the moving controls, remove the constraints targeting drivers)
2. delete the empties with the motion (leave stale constraints but shouldn't cause issues)
3. hard way: change the config, create a custom default for the future

3. Will do some more experimenting over the next week and report back.

Cool :)

4. Question: What is FACE_DATA - compared to FACE? Did notice some drivers make spots on the face move (which is good) but others don't.

Face data is the raw data from mediapipe - there's lots of stuff that isn't actually used in the transfer. I recognised that some drivers from the face have way to weak settings in the current config - planning to change that but atm setting up a socket to send data to blender via an external app. Hope I'll get the separate exe build soon - webcam tracking runs with 30-90fps which is pretty amazing compared to the 8-20 in blender.

[KaijuKoder]

TLDR: Face is useless without Pose (unless face stares forward and doesn't move much)

Testing

Did some more experimenting. Found when BlendARmocap works, it's brilliant. You can do in an hour an animation that would manually take a day, but it only handles a few cases optimally: Body pose only, and only static face on a head that looks straight ahead and doesn't turn, but anything else - or combining multiple - produces weird results which are so hard to clean up, faster to do it manually.

Merge Face/Pose/Holistic? I thought that's what holistic is for, I think I don't fully understand.

1. I think you're right about Holistic = Face + Pose (I thought Face had more facial datapoints than Holistic, but on closer look it doesn't seem to).

2. However when you go to transfer data, you can't do Holistic drivers, you must do Face and Pose separately. This is where it becomes difficult:

3. Chest/shoulders is in Pose, and Head + face is in Face. You need both transfers active at once, because Head doesn't really work without Pose: If the character doesn't move much, it's fine, but if they turn their head by anything but the smallest amount, you get Exorcist-like neck distortion. With a lot of mucking around (not easy, I was able to do it once, but couldn't replicate it) you can do multiple transfers to the same rig, but it'd be better to do it in one step. (It's hard to think of a use-case doing it in multiple steps is better.) Generally though I found doing a second transfer seems to erase the results of the first transfer? The UI isn't clear here.

I think the abstraction that's hard to grasp is that the data is just on empties, has something in-between and then gets copied by constraints. There are two easy ways of disabling: 1. remove the constraints from the rig (just go to the moving controls, remove the constraints targeting drivers)

It's really hard to spot the constraints (described below). Also not clear how multiple transfers are handled.

2. delete the empties with the motion (leave stale constraints but shouldn't cause issues)

It's messy and I found my rig was packed with many, many identical constraints, and the rig was giving really bad distortion, which made it all the harder to clean up. In the end I gave up.

3. hard way: change the config, create a custom default for the future

Alternate is to just delete them when they're not needed? Suggestions d/e.

4. There are problems with bone constraints (and the object constraint) on the rig. It seems doing multiple transfers causes problems, like adding duplicate constraints (same constraint seems to be added each time you do the transfer, not just on bones, but the object too: At one point I had 8 identical 'cgt_pose_location.D ' constraints on the object). Tried baking into a separate action and removing the constraints before a second transfer, but that's really hard and the 'Remove Constraints' option on Baking didn't remove them anyway) so tried doing it manually, but hard because so many of them, and some of them you must leave for Rigify. Sometimes it's easy to tell (if they have a target named "cgt" something empty), but many of the constraints seem to target regular bones making them hard to tell apart. Eventually I wrote a script to do this, but it's still hard to work out what constraints to remove, and noticed some of the target names changed between releases.

In the end concluded it unsafe to reuse the same rig for multiple BlendARmocap transfers, which prevents you from doing Pose+Face, and also prevents you from reusing an existing blend file to starting point for a new shot. That really slows the user down. (You can reset the override, but you still need to get the rig back into the starting posture). Another solution is to animate in a separate file so you can take out just the Action, but that makes it hard to set up the rig to match the scenery. Better if you can do it in one file, since the only reason for doing it in a separate file is to be able to get rid of the constraints.

5. Unless it's a full height video, Pose gives Exorcist-like full body distortion. Limbs which aren't in the video (say legs of a person sitting at a desk) seem to get glitching random orientation. It's very hard to work with. A newcomer to BlendARmocap would think it's completely broken. If you go constraint diving, you can clean it up, but there are so many of those constraints and it's hard to know what does what. I spent hours yesterday trying to figure it out yesterday. I'd suggest either (i) get Mediapipe not to move limbs which aren't visible, or (ii) simple checkboxes on the UI that lets the user enable/disable entire groups of constraints: Arms, Legs, also... Easier for the UI to do it than ask the user. Also a checkbox for...

6. Origin. Even though the character is in front of the camera, BlendARmocap/Mediapipe put them at really weird offsets from the origin for reasons I can't understand (7.78034893 meters forward, -3.3774 m underground, and 2.14489 m to the left: I made those numbers up, but that kind of thing.) Disabling the cgt_pose_location.D on the origin, and the trunk, and zeroing that seemed to work. But a checkbox to turn that off too would be good. Though the offsets seem very weird, so perhaps begin it from an origin anyway.

What I did find was if I got rid of the random limb flailing, if I could isolate the head and shoulders (and arms), I could get good results. But you can't see those good results until you've done multiple bakes and removed the constraints (which is hard to do).

7. Noticed limbs are still IK, not FK.

8. Did notice it doesn't seem to detect blinks. Those aren't hard to add manually, but obviously automatic is better.

9. Same it doesn't detect what the subject is looking at (pupils), but on Mediapipe's demo page you can see a demo of this and it looks nice. Again, not hard to animate manually, but obviously automatic better.

10. Noticed BlendARmocap only maps some of the Mediapipe's facial animation onto the face. It looks good, but more is better. This is where BlendARmocap shines. It's very hard to do that manually, and even if the little wobbles of a persons head and shoulders look great and also a lot of work to do manually.

11. Did notice some retargeting issues. Sometimes it works, but sometimes so far off faster to give up and do manually. Not just the arm length, shoulder width, but eye poking through facial skin etc. I've been reading on the math behind retargeting. It can be done, but it's a lot of work.

12. Original workflow I was using (described in an earlier post) was to keyframe just certain frames and manually inbetween, but realized that misses a lot of quality mocap. Realized it's better to isolate out the Exorcist-like artefacts (which is why I was using that - to get rid of the bad stuff, but realize it was getting rid of the good stuff too), and smooth what remains. Gives much better final results.

Suggestions:

a. Don't animate limbs which Mediapipe can't see in the video.

b. On the GUI, add a DRIVERS setting to transfer Everything at once.

c. Make clear on the GUI how multiple-transfers work (are they allowed? If so, warn prior transfers are overwritten... if they are? If multiple transfers aren't allowed, disable the button so the user isn't tempted to try. Thanks for explaining what FACE_DATA is, though a hint would help - because it looked to me like more detailed facial animation which it obviously isn't.)

d. Checkboxes to enable/disable constraints, also a button to remove them. Suggest this because it's hard to tell if the mocap worked, and risks scaring away newcomers who think it is broken because they don't realize once they get rid of the erroneous limbs there is a decent mocap hiding underneath. Something like this... (feel free to use).

    # Enabled: True, False, or None to toggle
    # Remove: True to delete constraint
    if rig.type == 'ARMATURE':
        for PB in rig.pose.bones:
            for c in PB.constraints:
                if getattr(c, "target"): # Need to check
                    # These constraints __seem__ to be BlendARmocaps. There are probably more... (SWITCH_PARENT?) Might be false positives here...
                    if c.target and (c.target.name.startswith('cgt_') or \
                                     c.target.name.startswith('eyebrow_in.') or \
                                     c.target.name.startswith('eyebrow_out.') or \
                                     c.target.name.startswith('lips.') or \
                                     c.target.name.startswith('lip.') or \
                                     c.target.name.startswith('lid.') or \
                                     c.target.name.startswith('head.')):
                        if Remove:
                            PB.constraints.remove(c)
                        else:
                            if Enabled is None: # If None we toggle
                                Enabled = not c.enabled

                            c.enabled = Enabled

e. Better yet, name all the constraints BlendArMocap adds new_constraint.name = 'BLENDARMOCAP ...', so it's easy to spot them.

f. A 'Clean Up' button to remove BlendArmocap drivers/constraints.

g. Suggest adding FK. I run this script fragment. Feel free to use:

def enhance_blendarmocap_to_fk(context, rig):
    assert rig.type == 'ARMATURE':

    blendarmocap_to_fk_mapping = {
    'upper_arm_fk.R':   'cgt_left_shoulder',
    'forearm_fk.R':     'cgt_left_elbow',
    'hand_fk.R':        'cgt_left_wrist',
    'thigh_fk.R':       'cgt_left_hip',
    'shin_fk.R':        'cgt_left_knee',
    'foot_fk.R':        'cgt_left_ankle',
    'upper_arm_fk.L':   'cgt_right_shoulder',
    'forearm_fk.L':     'cgt_right_elbow',
    'hand_fk.L':        'cgt_right_wrist',
    'thigh_fk.L':       'cgt_right_hip',
    'shin_fk.L':        'cgt_right_knee',
    'foot_fk.L':        'cgt_right_ankle'
    }

    for rigify_fk_bone, cgt_object in blendarmocap_to_fk_mapping.items():
        RPB = rig.pose.bones[rigify_fk_bone]
        CTO = bpy.data.objects[cgt_object]

        for c in RPB.constraints:
            RPB.constraints.remove(c)        

        con = RPB.constraints.new('COPY_ROTATION')
        con.target = CTO

        con.name = 'BLENDARMOCAP copy_rotation'  # so easier to spot when cleaning up Suggestion (e)

        print('Linked {rig.name} {rigify_fk_bone} to {CTO.name}')

h. Blinks and pupil movement is a nice to have for the distant future. The above is far more important.

i. Same with optimal retargeting. Nice to have but a lot of work.

j. Usability: Wondering if it might be nice to integrate the bake and remove constraints step into the BlendARmocap panel, as it's likely to be at the end of most people's workflows, and the options on the Bake window aren't very clear and many user's might not even realize finalizing with a bake is a good way to close.

What's the "live" version? The version from the main branch? Can you let me know the error message?

i. Sorry didn't have the log for the protobuf error. Not sure of Github's terminology, but it only happened when I tried to install what I think you call 'the main brach' (Code>Download from https://github.com/cgtinker/BlendArMocap). When I installed Latest from https://github.com/cgtinker/BlendArMocap/releases, worked fine.

j. Very Minor bug: If you accidentally select a wrong or unsupported filetype as the Movie, it gives you a confusing error 'OSError: Cannot open webcam'.Would be nicer if it said 'unrecognized video file type'.

k. Productivity idea: Also consider a 'Sitting still' and 'Standing still' checkbox. I guess Mediapipe has this tendency to pose characters bending forwards. If you have those checkboxes, can correct the bend so character's back is straight, legs are in sitting pose, etc. Spares the user the effort of constraint hunting or having to correct manually. 90% of shots will probably be a character just standing still or sitting.

Summary

BlendARmocap has a lot of potential. It is close. When it works well, it gives a staggering fast workflow. Biggest problems are shoulder/head+face separation, and managing the constraints.

Late Update:

Did some more experimenting, doing and baking different transforms in different blend files, then appending those into a single blend file. Result is really good and nice!, and it sidesteps the issues caused by lingering constraints. So conclude because of the above, you can't currently safely do multiple transforms in a single blend file, yet multiple transforms are necessary because even the minimum use case of a "talking head" needs both the pose and face transfers, otherwise you get that Exorcist-like neck twisting artefact.

[cgtinker]

Thanks for your suggestions! Let me know what you think about the following:

1. I think what's kinda confusing is, that you basically just transfer once, you don't have to transfer twice or multiple times. Also, transferring everything at ones works by basically just transferring the parent collection (cgt_DRIVERS). Probably I should make a drop down which ensures that a little more.. also prolly add a custom property to rigs which indicates what has been transferred already (or something like that) so I don't recklessly overwrite / add doubles.
2. What happens under the hood is the calculation of rotations and stuff and then the raw and calculated data gets in the scene - there is no real connection between the data and the confidence values - so probably adding a "sitting / standing" flag is a good idea. Not really sure what's the best way to do that as everything is based on configs which just use the data from collections (it's object name based, which sucks but there are not static id's on blender objects so there aren't a ton of options) you can take a look at the current basic config BlendArMocap/src/cgt_transfer/data /Rigify_Humanoid_DefaultFace_v0.6.1.json if you like. Such configs can be created / modified with my.. well ui in the constraint setting when selecting an empty generated by blendarmocap. It works okay but I'm not super happy with it...
3. I kinda like the idea of configs and hope I can reuse them. Those are just large json which contain all the data. So all constraints / driver settings of a config etc is in there. So even when drastic changes happen I can prolly build on top of them.
4. you are totally right about this weird offset by the holistic pose but that's basically just what mediapipe detects. I could zero the start position which would kinda make sense. I kinda made a tradeoff between keeping the data as close as possible to the detection results while making it useful for animation. Guess in the future I'll auto smooth a little more aggressively. Will see
5. Guess I'll update the face config and add a config for FK transfer - will make live a little easier for some. IDK about flags for legs etc.. It kinda makes sense but it kinda doesn't work with the current system. The config system idea is so it can basically be used with any control rig and not just the rigify humanoid rig.. Seems kinda out of scope atm. However, prolly such a flag is like a preset, so legs don't even get recorded - that kinda would make sense...
6. True there are no eyes, still a thing for the future. This will be a thing for the external executable IF I can establish a stable build pipeline. The dependency setup in blender is to unstable to be supported in the future.

At the moment blender is planning to actually create a control rig system. The current approach with blendarmocap is kinda hacky as blenders current system isn't really designed for the stuff I'd like to do with it. I've started joining blenders module meeting for animation and rigging - plan to show my stuff and trying to take part on the integration of the new system somehow (if the let me :D) - I think it makes more sense on building on top what will be used and making the integration not to be a hack. Otherwise I'd have to rework everything anyways (in approx 1.5 years)...

[KaijuKoder]

I think what's kinda confusing is, that you basically just transfer once, you don't have to transfer twice or multiple times. Also, transferring everything at ones works by basically just transferring the parent collection (cgt_DRIVERS). Probably I should make a drop down which ensures that a little more.. also prolly add a custom property to rigs which indicates what has been transferred already (or something like that) so I don't recklessly overwrite / add doubles.

LOL. I didn't even realize you could transfer all-at-once by choosing 'cgt_DRIVERS'. I assumed that meant 'Pick one of these drivers'. Come to think of it some renaming would make things a lot clearer+easier. 'Everything' instead of 'cgt_DRIVERS'. And 'Everything' or 'Head+Body+Hands' instead of 'Holistic' which is a vague+confusing term the Medidapipe team chose. Maybe 'Body only' instead of 'Pose' too. 'Pose' by itself doesn't mean much.

Constraint management would be greatly appreciated. Just give every constraint added a 'BLENDARMOCAP ...' name, and it'll be easy to control.

What happens under the hood is the calculation of rotations and stuff and then the raw and calculated data gets in the scene - there is no real connection between the data and the confidence values - so probably adding a "sitting / standing" flag is a good idea. Not really sure what's the best way to do that as everything is based on configs which just use the data from collections (it's object name based, which sucks but there are not static id's on blender objects so there aren't a ton of options) you can take a look at the current basic config BlendArMocap/src/cgt_transfer/data

IDK about flags for legs etc.. It kinda makes sense but it kinda doesn't work with the current system. The config system idea is so it can basically be used with any control rig and not just the rigify humanoid rig.. Seems kinda out of scope atm. However, prolly such a flag is like a preset, so legs don't even get recorded - that kinda would make sense...

Two parts to this. First, the legs/arms/origin checkboxes change nothing of the config. Only which rig constraints are currently enabled/disabled via toggling. I don't think I've described it well, so let me rephrase it: Imagine you are a user who has done a transfer and your rig looks like an Exorcist abomination (erroneous twisted flailing limbs, thanks to Mediapipe's weird handling of off-screen/undetectable limbs). If you click the legs checkbox, it runs through the rig and disables the BLENDARMOCAP-created constraints related to the legs: foot_ik.L, foot_spin_ik.L, shin_tweak.L, etc (I'm not really sure which ones they are, but you get the idea?). Suddenly instead of the charcters legs flailing widly, the rig is now standing upright - moving except for its legs. Didn't want that? Click the checkbox again, and it re-enables them.

What this does is let you quickly+easily isolate erroneous+unwanted limbs out of the rig; something Mediapipe has a horrible habit of doing. It makes for a much faster workflow.

Second part: Sitting+standing flags. Idea there was it would correct the erroneous forward lean Mediapipe likes to do. Would be nice to fix that at the outset, but sounds like a lot of work, so user can correct later in the NLA instead.

/Rigify_Humanoid_DefaultFace_v0.6.1.json if you like. Such configs can be created / modified with my.. well ui in the constraint setting when selecting an empty generated by blendarmocap. It works okay but I'm not super happy with it... I kinda like the idea of configs and hope I can reuse them. Those are just large json which contain all the data. So all constraints / driver settings of a config etc is in there. So even when drastic changes happen I can prolly build on top of them.

Had a look at the config file. It looks daunting, but like most things I guess, once you understand it, it becomes easy.

you are totally right about this weird offset by the holistic pose but that's basically just what mediapipe detects. I could zero the start position which would kinda make sense. I kinda made a tradeoff between keeping the data as close as possible to the detection results while making it useful for animation. Guess in the future I'll auto smooth a little more aggressively. Will see

Understand, but no need to add more smoothing; Again suggest those checkboxes. Have an Origin checkbox which toggles enable of the rig object and torso bone cgt_pose_location.D copy location constraint. Do that and your rig is suddenly back at the origin. Problem solved. (Change your mind? Click the Origin checkbox again, constraint is re-enabled, goes back to mediapipe's offset origin).

When you bake, the disabled constraints are excluded from that. Works nicely.

Guess I'll update the face config and add a config for FK transfer - will make live a little easier for some.

I found it really helpful.

True there are no eyes, still a thing for the future.

Yes. One day.

This will be a thing for the external executable IF I can establish a stable build pipeline. The dependency setup in blender is to unstable to be supported in the future. ... I think it makes more sense on building on top what will be used and making the integration not to be a hack. Otherwise I'd have to rework everything anyways (in approx 1.5 years)...

The dependency pipeline is a nightmare many Blender addons, anything Python really, and also the Linux dynamic link libraries. (Agree with this guy https://lore.kernel.org/lkml/CAHk-=whs8QZf3YnifdLv57+FhBi5_WeNTG1B-suOES=RcUSmQg@mail.gmail.com/ ) Caused by 'Move fast and break things' developers who make arbitrary and often unnecessary API changes which break everything downstream. I don't know why they do it. It's why there is are hundreds of broken Blender addons strewn along the roadside. Sadly don't think much can be done about it, and I'm yet to see a docker type system which works well.

On the bright side, BlendARmocap is working today! :-)

At the moment blender is planning to actually create a control rig system. The current approach with blendarmocap is kinda hacky as blenders current system isn't really designed for the stuff I'd like to do with it. I've started joining blenders module meeting for animation and rigging - plan to show my stuff and trying to take part on the integration of the new system somehow (if the let me :D) -

Nice! Blender's animation system badly, badly, badly, needs an overhaul. But they keep delaying it to add shiny bells and whistles which no one ordered. I just wish they'd give us a decent animation system. Trouble is there's not much else. Unreal maybe.

On an upbeat node. Once I got past the hurdles yesterday, I was able to do two scenes yesterday. Results were really, really, nice. You've achieved something quite amazing with BlendARmocap. It gives nice results, quickly, on your own PC! You don't need an expensive glitchy mocap suit, nor using a web site which 'retains' your IP and with lengthy uploading delays. BlendARmocap becomes a killer app for Blender. Hope the Blender devs realize that.