Possible to rig face animations with webcam?

[yeemachine]

Really glad to see someone working on a Live2D Pixi plugin that might see updates in the near future. Having the character follow a cursor point seems to work well.

I am just getting into Live2D, but is there a way to control the avatar/animations even more precise? For example, if it was hooked up to some webcam facecapture, can the plugin tap into more specific controls like mouth/eye movements?

[guansss]

It's nice to see this project being useful!

Yes, it's possible to manipulate the internal parameters by some low-level methods, that is, setParamFloat(), addToParamFloat() and multParamFloat(), see declaration file.

I believe some key parameters are common to all Live2D models, such as PARAM_MOUTH_OPEN_Y and PARAM_EYE_L_OPEN, you can find them out by inspecting Live2D models in Live2D Viewer.

[yeemachine]

I've played with the rigging points atm with faceapi.js so I should be able to detect the distance between, let's say, of opening and closing of the mouth. How would I then interpolate that value into the Live2D model?

model.focus seems to be where I can just drop in an x,y coord. A bit confused on how to adjust paramaters for mouth open/close. Let's say I wanted a simple interpolation of 0-1 open/close.

Here's a demo link with rough webcam tracking, https://live2d-cam.glitch.me/ Nothing too specific, still using your demo model as a test, I just logged the model obj for reference. Source

[guansss]

It's a bit more complicated than I was thinking, all parameters seem to be reset to specific values when running motionManager.update(), no matter if there's a motion playing or not.

So the parameters should be overwritten after calling that function. Parameter related to the mouth is PARAM_MOUTH_OPEN_Y.

const updateFn = model.internal.motionManager.update;

model.internal.motionManager.update = () => {
  updateFn.call(model.internal.motionManager);

  // overwrite the parameter after calling original update function
  model.internal.coreModel.setParamFloat('PARAM_MOUTH_OPEN_Y', mouthValue);
}

Somewhat tricky, there may be better solutions for production.

Here's the demo.

[yeemachine]

Oh nice! thanks for putting this together. Definitely feels more clear on how to operate around the Live2D motions. I'm not too sure what that extra updateFn declaration does to the function, but overwriting of the parameter still works if I comment that line out. Unless I'm missing something?

model.internal.motionManager.update = () => { model.internal.coreModel.setParamFloat('PARAM_MOUTH_OPEN_Y', mouthValue); }

Here's an updated fork of it

[guansss]

The update function updates all parameters by the motion, without calling this function the model will not be able to play motions. In the original demo, you can see some small movements on the model like turning head and changing the mouth form between :D and :O, this is the effect of idle motions, which disappears in the forked version.

[yeemachine]

Oh, got it! Thanks for figuring out how to add custom params to the models. Going to have fun mapping the motions. I'll keep checking back in case you do update the library to support Cubism 4 or any other quality of life updates.

[yeemachine]

Sorry, one more question. So I've started to stack multiple manual animations on, but the PARAM_EYE_L_OPEN and PARAM_EYE_R_OPEN seem to freeze and lock to full stares at 3 blinks and then freezes. Perhaps some there's an animation cycle that's breaking the manual params inputs? It seems to be the same when switching any of the random expressions.

Demo

[yeemachine]

Nevermind. Guess the Live2DEyeBlink class overrides whatever params you set directly. model.internal.eyeBlink.setEyeParams(mouthValue). Demo now works properly.

[thisismygatekeeper]

@yeemachine @guansss thanks for the tips! Just tried model.internal.eyeBlink.setEyeParams(eyeValue) but for some reason it doesn't work for v4 models. Could you suggest some other possible controls? Thank you!

[ye7iaserag]

@guansss long shot here but have you tried this hack with the latest version? whatever I do the mouth Y value gets reset

[ye7iaserag]

Ok got it working with cubism4, here is the code:

const updateFn = model.internalModel.motionManager.update;
model.internalModel.motionManager.update = () => {
    updateFn.call(model.internalModel.motionManager, model.internalModel.coreModel, Date.now()/1000);

    // overwrite the parameter after calling original update function
    let mouthValue = Math.sin(performance.now() / 200) / 2 + 0.5; // mimic the interpolation value, 0-1
    model.internalModel.coreModel.setParameterValueById("ParamMouthOpenY", mouthValue);
}

[NEKOparapa]

Ok got it working with cubism4, here is the code:

const updateFn = model.internalModel.motionManager.update;
model.internalModel.motionManager.update = () => {
    updateFn.call(model.internalModel.motionManager, model.internalModel.coreModel, Date.now()/1000);

    // overwrite the parameter after calling original update function
    let mouthValue = Math.sin(performance.now() / 200) / 2 + 0.5; // mimic the interpolation value, 0-1
    model.internalModel.coreModel.setParameterValueById("ParamMouthOpenY", mouthValue);
}

Thank you, this piece of code is very useful

[Naozumi520]

Ok got it working with cubism4, here is the code:

const updateFn = model.internalModel.motionManager.update;
model.internalModel.motionManager.update = () => {
    updateFn.call(model.internalModel.motionManager, model.internalModel.coreModel, Date.now()/1000);

    // overwrite the parameter after calling original update function
    let mouthValue = Math.sin(performance.now() / 200) / 2 + 0.5; // mimic the interpolation value, 0-1
    model.internalModel.coreModel.setParameterValueById("ParamMouthOpenY", mouthValue);
}

Yes it did works, but I got this error with calling it few times:

uncaught RangeError: Maximum call stack size exceeded
at model.internalModel.motionManager.update (index.js:25:52)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)
at model.internalModel.motionManager.update (index.js:26:22)