Make avatar eyes look at participant

[timjzee]

Something like this: https://www.youtube.com/watch?v=NZmQPSKf7YY

But use Poses from mocap instead of Maya.

[timjzee]

Or this: https://forums.unrealengine.com/t/anyone-know-how-i-could-do-this-with-a-metahuman/554928/3

Seems more elegant

[timjzee]

https://docs.unrealengine.com/5.3/en-US/control-rig-in-animation-blueprints-in-unreal-engine/

[timjzee]

This works if we also enable CTRL_lookAtSwitch, and we need transform world location to the location of the CTRL_C_eyesAim control.

Control Z = World X Control X = World Z Control Y = World Y + 20

Use Inverse Transform Location on the Focus location: https://forums.unrealengine.com/t/converting-a-vector-from-world-space-to-local-space-and-vice-versa-in-blueprint/302441/2

[timjzee]

Next steps:

• implement world to local transform
• Test if it works at world origin
• Test if it works at different world locations and orientations (make sure CTRL_C_eyesAim is child of root in Control Rig)
• Test if it works if we also have another animation playing
• Make it work with VR
• Implement intervals of eye contact

[timjzee]

Getting this working required modifying the Control Rig like this

Also eyes should be set to converge to prevent weird asymmetries between eyes.

Next Steps:

• Test if it works while body and facial animations are being played.
• Adjust AnimationBP to focus on VR Pawn camera instead of the Focus Ball.
• Implement intervals of eye contact.

[timjzee]

Getting this to work while an animation is being played turns out to be very difficult. It should be doable using a "Layered blend per bone" with a Branch Filter or Blend Mask that specifies which bones should be blended between the animation and the eye aim. However, this doesn't work for some reason:

• https://forums.unrealengine.com/t/how-to-blend-metahumans-facial-animation/754799
• https://forums.unrealengine.com/t/metahuman-facial-animation-blend-mask/1228532
• https://forums.unrealengine.com/t/ue5-is-it-possible-to-layer-animations-per-bone-on-a-metahuman-face-alternatives/558337

I have figured out a workaround that involves:

• Creating an Upper Face and Lower Face version of each facial animation sequence
  • For the Upper Face version we select all controls that do not control eyes or brows > Edit curves > Drag to select all data points > Right-Click > Delete
  • For the Upper Face version we select all controls that do control eyes or brows, and again delete all data points for those curves.
• We then blend (using Layered blend per bone and selecting "Use Max Value" for the "Curve Blend Option") Lower face animation and animation driven by the Control Rig whenever we want eye contact, and blend Lower face and Upper face animations whenever we want the original eye/brow animation.

However, this probably also requires the facial animation to also include the head rotation. Let's try that.

[timjzee]

something to look at: https://www.reddit.com/r/unrealengine/comments/ngtc2w/control_rig_outputs/ https://docs.unrealengine.com/5.3/en-US/control-rig-in-animation-blueprints-in-unreal-engine/ https://docs.unrealengine.com/5.3/en-US/animation-blueprint-blend-nodes-in-unreal-engine/#layeredblendperbone

May be useful for debugging: https://docs.unrealengine.com/5.3/en-US/BlueprintAPI/Components/SkeletalMesh/GetSocketLocation/ https://docs.unrealengine.com/5.0/en-US/skeletal-mesh-sockets-in-unreal-engine/

[timjzee]

Problem: Eye aim does not consider rotation of the head.

Things to try:

• see if blending just the body animation and the control rig adjustments works
• see if exporting facial animation with alternative head movement option works
• see if feeding the body animation into the control rig adjustments works
• try this method to get head movement working: https://www.youtube.com/watch?v=Rym1QpD_Wr4

[timjzee]

Solution ended up being: feeding the body animation into the control rig adjustments