Same anim blueprint with different skeletal meshes breaks "Add To Existing" blending

[dpakatheman]

Currently, the FaceFX UE4 plugin allows something similar to UE4's "animation retargeting" by changing the properties of the "Blend FaceFX Animation" node in the Animation Blueprint from "Replace Existing" to "Add to Existing". This allows the same FaceFX asset to be used with different animation blueprints as long as the skeleton is similar enough to use offset transforms. Offsets are calculated by subtracting the reference pose of the skeleton from the FaceFX output transforms every frame to implement the "Add to Existing" behavior.

The problem occurs when the same animation blueprint is used with different skeletal meshes that you want to play retargeted animations on. The attached reproduction case demonstrates this (a big thank you to our customer for submitting this). The wrong reference pose is getting subtracted from the transforms in this case.

FaceFX_Test.zip

The FaceFX Runtime supports offset transforms by passing FFX_USE_OFFSET_XFORMS instead of FFX_USE_FULL_XFORMS when calling ffx_create_bone_set_handle (in FaceFXCharacter.cpp). The FaceFX UE4 plugin does not utilize this feature currently, but it would be faster than subtracting the reference pose every frame, because the computation would only done once when the bone set handle is created. It would also fix this problem since the skeleton's reference pose would never need to be subtracted.

It is relatively easy to fix this issue if your game ALWAYS wants to use offset transforms and you will always specify "Add to Existing" in your Blend FaceFX Animation node. Simply pass FFX_USE_OFFSET_XFORMS instead of FFX_USE_FULL_XFORMS in FaceFXCharacter.cpp, and remove the code in AnimNode_BlendFaceFXAnimation.cpp that subtracts the reference pose of the skeleton.

The more difficult case is to support games that want some combination of "Add to Existing" and "Replace Existing". It is theoretically possible to have the same FaceFX asset being used by two different Blend FaceFX Animation nodes with different configurations. One possible general-purpose solution is to always load bone set handles twice, once with FFX_USE_OFFSET_XFORMS and once with FFX_USE_FULL_XFORMS, so that the correct transform is always available at runtime. I am unaware of a use case that requires a combination of blend modes, so a simpler approach may also be possible.

[dpakatheman]

Some important notes about this change:

1. A new Project-wide setting (Edit->Project Settings->Plugins->FaceFX Game->Default Blend Mode) now controls if you want FaceFX to work in "Add to Existing" or "Replace Existing" mode.
2. The per-project setting can be over-ridden on a per FaceFX Actor file basis. Right-click on the FaceFX Actor object, then go to "Asset Actions->Bulk Edit via Property Matrix"
3. Additive blending is now more performant
4. Additive blending now supports cases where the same animation blueprint is used on skeletal meshes with different skeletons
5. "Add to Existing" and "Replace Existing" settings for Position, Rotation, and Scale have been removed from the FaceFX Blend node in the Animation Blueprint. It is no longer possible to have different settings for Position/Rotation/Scale components.
6. In the obscure case when you want the same FaceFX Actor to work with "Replace Existing" sometimes and "Add to Existing" other times, you must create a duplicate FaceFX Actor so you can apply different settings via the Bulk Edit Property Matrix

Note that additive blending works by subtracting the FaceFX rest pose (typically this is frame 0 of the FBX file you dragged onto FaceFX Studio) from the bone poses to create the additive poses. The additive poses are then blended onto your skeleton, making minor changes in the skeleton's reference pose less important. Be aware that the FaceFX rest pose is not necessarily the same pose as the UE4 skeleton's reference pose or the bind pose of the Skeletal Mesh (but it could be). If you plan on using the "Add to Existing" feature for animation retargeting, it is important to understand that this is how we calculate what is "added" to your skeleton.

[marckury]

hi , its 2021 in metahuman times , and i was trying to use some of this add poses or animations stuff but with no luck ...... metahuman have this face post animation blueprint where all bones are locked blocked and even adding some animation to offset them dont work , so i wonder if this plugin or some workaround to can offset metahuman bones after this blueprint , i have same skeleton and face but with some bones animated to get different pose , i just want add this animation as offset to the origginal metahuman , any idea? thx!

[dpakatheman]

The best way to drive metahumans with FaceFX is to create FxAttribute nodes with names like:

CTRL_expressions_eyeLookRightR
CTRL_expressions_browDownL
CTRL_expressions_jawOpen

Drive those with animation data from FaceFX, export an FBX file (a simple skeleton with nothing but a root bone will work here as the input FBX), then import it into UE4 specifying the "Face_Archetype_Skeleton" as the skeleton under the mesh section.

You won't have much luck driving metahumans using the FaceFX-UE4 plugin because the actual bones/morphs change based on the LOD and metahumans needs to generate the "riglogic expression curves" from the input.

[marckury]

thx for the reply! even if not using this plugin , experienced unreal people dont have any other solution? the thing is im maya user , and i see in the blueprint the final control rig , after that can move joints freely with a transform bone , i was wondering if theres some kind of coding way to move all joints to a desired location at once or faster , because i can do that in maya , but cant find a way in unreal , in some web i saw this toggle disable post animation blue print that could help to move bones and dont be blocked by the dna file but i cant find it in my unreal project , anyways thx

[dpakatheman]

Sorry, I don't have an answer for you. Metahumans are more than fancy characters...they include technology that enable the high quality results and if you have questions about this technology and how it interacts with your desired workflow, this isn't the right place to get those questions answered.