aws-samples / amazon-sumerian-hosts

Amazon Sumerian Hosts (Hosts) is an experimental open source project that aims to make it easy to create interactive animated 3D characters for Babylon.js, three.js, and other web 3D frameworks. It leverages AWS services including Amazon Polly (text-to-speech) and Amazon Lex (chatbot).
MIT No Attribution
175 stars 80 forks source link

Speech animation, gaze tracking, face shape break with BabylonJS v5 #157

Open Krxtopher opened 1 year ago

Krxtopher commented 1 year ago

Description:

If you attempt to use the Hosts with BabylonJS v5, audio and body animations work properly, but there is no speech animation, gaze tracking stops working, and the shape of the characters head (jaw, chin, and eyes especially) look odd (see attached images.)

Steps to Reproduce:

To reproduce:

  1. Clone the respository.
  2. Ensure you have branch “mainline2.0” checked out.
  3. Delete the package-lock.json file.
  4. Open the file /packages/amazon-sumerian-hosts-babylon/package.json for editing.
  5. Change the “peerDependencies” to the following:
    "peerDependencies": {
    "@babylonjs/core": "^5.0.0",
    "@babylonjs/loaders": "^5.0.0"
    }
  6. From the root of the repository (important), run npm install
  7. Edit the /demo-credentials.js file to use a Cognito Identity Pool you’ve set up for use with Sumerian Hosts (see /AWS-Infrastructure-Setup.md for instructions if you don’t already have this set up.)
  8. From the root of the repository, run npm run start-babylon
  9. Once the browser tab titled “BabylonJS Sumerian Host Demo” loads, select the “Hello World Demo”.
  10. Click the “Start” button on the demo.
  11. Observe that the that audio plays but the host’s mouth doesn’t move and the host’s jawline/chin/eyes don't look right.

Expected:

Host w BJS v4 2 1

Actual:

Host w BJS v5 - front Host w BJS v5 - profile
trusktr commented 4 months ago

duplicate of #155