emilianavt / VSeeFaceSDK

The VSeeFace SDK for Unity is used to export models in the VSFAvatar format.
149 stars 10 forks source link

Combo expressions problem #18

Closed katrinshtorm closed 1 year ago

katrinshtorm commented 2 years ago

Hello...When using vrchat there is an ability to blend animations of facial expressions. I wanted to set facial expressions the same way, so they could blend together when using experimental expression detection. I've recorded a video(link below). The problem is when I record expressions on Fun/Angry and other they don't combine when I activate 2 expressions. I assume this is because if you set face value in blendshape editor then together with active expression blendshapes, passive nonactive blendshapes is also activated - that's why they don't combine. The same way animation work in vrchat. Because the method abouve didn't allow me to combine expressions, I decided to try another way. VSF allows you to use script - VSF_Animations. I've recorded needed animations with ability to combine them and applied script to them. However... in VSeeFace I can't activate animation of face expression. I activate recording key with correct expressions(smile), record, but after I use Calibrate expression - I can't activate it, but it's recorded(if expression is not made with animation but in blendshape editor then it activates correctly). I've recorded a video where I'm trying to record activating animation-expression for Joy. Also I'm including video where I'm combining facial expressions using keyboard. With keyboard you can activate multiple type of facial expression, but with Expressions detection you can't

Combo expressions problem VSeeFace Smile Custom Animation(included in vsf_animations script) recording Combining expressions (using keyboard)

emilianavt commented 2 years ago

The expression detection function can only activate one of its expressions at a time, so having a blend between them is just not possible due to how it works. It's an SVM multiclass classifier that will output the most likely expression for a given frame.

It's possible using hotkeys, as long as the expressions are set to "Additive" in the expression settings.

katrinshtorm commented 2 years ago

All my expressions(except body animation) are set to Additive, but blend doesn't happen. In first video(Combo expressions problem) you can see clearly how 1 animations overlap the other

emilianavt commented 2 years ago

In the first video, expression detection is used, which cannot possibly detect multiple expressions at the same time due to the way it works. Basically, internally it returns a number from 0 to 6, each corresponding to one of the expressions and activates the one it returns.

katrinshtorm commented 2 years ago

You mean at the present time it's impossible to combine expressions in "experimental expressions" mode? Without hotkeys but with facial expression only.... sorry, english is not my native language

emilianavt commented 2 years ago

Yes, exactly.