does multi track animation support in ofAssimpModelLoader?

[ggslayer]

Hi, In my scene, I have a fbx model which make a human with two animations: "idle" and "mouse open", the human do idle animation looply, concurrently when the human need to speak, I start play the "mouse open" animation.

This is common reqirement in digital human business. In this situation, I think when mouse open animation playing, corresponding mouse skeleton is affect by only "mouse open" animation? I'm not sure for tech implement detail. In before I done this in 2d digital human app which used a library called spine2d, In which, have a freature called animation track, the deatil is here: https://esotericsoftware.com/spine-applying-animations. I think this can help me to illustration what I want.

So what I want to know Is: does the ofAssimpModelLoader addon support this feature? can you give me some advice about my question?

[NickHardeman]

Currently ofxAssimpModelLoader does not support per skeleton animations. I have added the feature request to a TODO checklist in this discussion regarding updating ofxAssimpModelLoader.

7722

[ggslayer]

Hi, I think..., To avoid different understanding to my reqirement, I want to make a description about the key points of my understanding.

1. Multiple animations can play concurrent. for each animation, may affect only subsets skeleton or whole skeletons.

2. When the multiple animation playing, each animation can change animation parameter independent, for example: animations runs in different speed, (in the digital human secen, when the human walking in normal speed, the mouse open and close speed may be faster or slow down affect by tts/asr result).

3. For tech implements, every sketelon is affect by only the "higher piror animation" like spine2d's animation track index value OR multiple animation 's weight value? That's all ok for me. because, if affect by multiple weight value, I will set the "mouse open" animation's weight is 1.0 and "walk" animation is 0.0, so, the "mouse part mesh" controll by"mouse open" animation, and other mesh controll by "walk" animation, which is same as spine2d's track index behavior.

4. If "animation track" concept exists, It's more comfortable，there may be multiple tracks. 4.1 In every track, has multiple animations, the animation in it will be play by loop or once, animations can switch by transaction with a given mix time param or switch no transaction. 4.2 Animations in different track playing concurrently , sketelon affect behavior runs like point 3.

So, I can done code like this:

auto animTrack1 = AnimationManager::CreateTrack("trackIdle", ...);

auto animationWalk = ofGetAnimation("walk");

// walk loop
animTrack1.addAnim(animationWalk, 0).  /*0 is loop, 1 is once, 2 is twice...*/

AnimationManager::Create("trackSpeak", ...);

AnimationManager::RunAllTracks();    // currently , the trackSpeak is empty.

// when I receive speak command, the code like this:
auto animTrack2 = AnimationManager.getTrack("trackSpeak");
auto animationMouse = ofGetAnimation("mouse open");
int64_t duration = animationMouse.GetDuration();
float speed = tts_mouse_time(...) / duration;
animationMouse.SetSpeed(speed);

// play once
animTrack2.addAnim(animationMouse, 1);

// in these code, there is no weight value set,  the skeleton include by multi animation is affect by the higher track index's animation( in this example is trackSpeak's animation is piror)

like the codes , the human can speak while walking, :).

I wrote these, submit a idea for ofAnimation higher concept and code example, only for describe my reqirement more clearly, avoid differents understanding.

Thank you for your patience.

[ggslayer]

After some careful thinking, I think I made a mistake in the implementation of speak before. speak is not simply to open mouth and shut up, but to map the phonemes of tts to the viseme pose animation, and then make transition between two pose animations to complete speak word. Therefore, the key point is the support of "pose to pose animation". In order to describe my requirement more specifically, I still write a pseudo code:

auto animTrack1 = AnimationManager::CreateTrack("trackIdle", ...);
auto animTrack2 = AnimationManager::Create("trackSpeak", ...);
auto animTrack3 = AnimationManager::CreateTrack("trackGesture", ...);

auto animationWalk = ofGetAnimation("walk");

// walk loop
animTrack1.addAnim(animationWalk, 0).  /*0 is loop, 1 is once, 2 is twice...*/

AnimationManager::RunAllTracks();    // currently , the trackSpeak is empty.

// when I receive speak command, the code like this:
auto phonemeInfos = parseTTSResultToPhonemes();  // phonemeInfo inlucde phoneme id / time pair

for(int i = 0; i < phonemeInfos.size(); i++){
    // get viseme pose animation id from config map, the map is fixed in config file.
    String poseAnimationName = getVisemePoseAnimationByPhonemeID(phonemeInfos[i].id); 

    int transitionDuration = 0;
    if(i != 0){
        transitionDuration = phonemeInfos[i].time - phonemeInfos[i-1].time;
    }

    // add the phoneme's pose animation to track, transition to the this anim from prev pose anim by transitionDuration.
    animTrack2.AddAnim(poseAnimationName, 1, transitionDuration);
}

// run a happyness gesture animation by mood analysis
float happyTime;
bool bHappy = IsHappy(ttsResult, &happyTime);
if(bHappy){
    auto happyGestureAnim = ofGetAnimation("happyness_gesture").

    // happyTime is the anim's start time from now
    // 0.5 is the transition duration from track3's prev anim to this anim  if prev anim is playing when happyTime reached;
    animTrack3.AddAnim(happyGestureAnim, 1, 0.5, happyTime);  
}

It would be great if ofassimp plugin could support the feature of pose to pose animation by transition. Of course, the multi track feature is still very important. Sorry to bother you. @NickHardeman

[NickHardeman]

Hello @ggslayer, I think you might be referring to skeleton retargeting and skeleton to skeleton animation blending with weights per bone for both animation and skeletons. Maybe something like described in the following links. https://docs.unrealengine.com/5.3/en-US/animation-retargeting-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/retarget-manager-in-unreal-engine/

There are a lot of edge cases to make a general use case skeleton retarget system, including, but not limited to, different number of bones, different scales, different axes orientations, bone name mappings, etc.

This is quite a large subject and out of the scope of what I plan to work on for the add on. With the current state of the add-on, it is possible to grab bones and programmatically control them so implementing bone retargeting is possible. Animation mixing per bone is already implemented, though not heavily tested. https://github.com/NickHardeman/openFrameworks/tree/addon-ofxAssimp/examples/3d/ofxAssimpBoneControlExample

[ggslayer]

Hi, I read your link about retargeting animation, and I didn't fully understand it, but I thought maybe it wasn't what I needed. I probably read the document of ue animation. What I needed was the so-called "Additive Animation" in the ue engine, which is very suitable for facial expressions and lip animation. After some thinking, I think there are two main points:

1. curve transition between base pose animation and target pose animation, this transition process forms a new animation A, which only stores the difference between the two poses, with the emphasis on "difference".
2. In the process of fullbody animation, every frame of animation A can be added to every frame of fullbody animation, with the emphasis on "add". There are several ways to add:

2.1 simple position addition, which is what I need. 2.2 Blend based on weight. 2.3 ...

Because the "difference" is added, the face will follow the position of fullbody, which is very important.

Reference link: https://docs.unrealengine.com/4.27/en-US/AnimatingObjects/SkeletalMeshAnimation/AnimPose/ https://docs.unrealengine.com/udk/Three/AdditiveAnimations.html

Please help me read my mind, is it right? It's best if you ofassimp can support it, If not, can you give me some implementation ideas and specific guidance base on your current work? (I'm just getting started with 3d graphics programming) Thank you very much. @NickHardeman

[ggslayer]

I see "Animation mixing per bone is already implemented", is that means "multi animations play concurrent with weight blend is ok" ?, I think I need two more features:

1. support of "base pose to target pose curve animation ", and make it as a "additive animation".
2. animations play concurrently blend with "add".

I want to know is these feature will be supported? if not, I will try to do it myself, for me, this maybe a big work :(

[NickHardeman]

Hmm, I am still a bit confused as to what you might ultimately want to achieve and how how it might be implemented. Have you taken a look at the bone control example? https://github.com/NickHardeman/openFrameworks/tree/addon-ofxAssimp/examples/3d/ofxAssimpBoneControlExample

You can blend animations per bone or entire skeletons. I am unsure if and how Assimp stores poses. The bones are imported at certain positions and orientations before the animations are applied. Not sure if that is considered a pose in other 3D software.