search

aws-samples / amazon-sumerian-hosts

Amazon Sumerian Hosts (Hosts) is an experimental open source project that aims to make it easy to create interactive animated 3D characters for Babylon.js, three.js, and other web 3D frameworks. It leverages AWS services including Amazon Polly (text-to-speech) and Amazon Lex (chatbot).
MIT No Attribution
175 stars 80 forks source link

Add support for BabylonJS v5 #155

Open Krxtopher opened 1 year ago

Krxtopher commented 1 year ago

The current implementation of Hosts is only compatible with BabylonJS v4 (4.2.1). When trying to use Hosts with BabylonJS 5 and later things break - like host speech animations and gaze tracking.

Krxtopher commented 1 year ago

This bug is related: https://github.com/aws-samples/amazon-sumerian-hosts/issues/157

sevaseva commented 1 year ago

Tracking this. May need this at some point. Babylon.js 6.0 ~will be~ released March 30, 2023, too. Thanks!

GeneUNCG commented 1 year ago

Would it be possible to change this to v6.0 now that it's released? My team is hoping this happens. Faster rendering would greatly improve our load times for our web-application.

R1ftwarr commented 11 months ago

What's the status on this? We have found sound is broken on iOS Safari in Babylon 4.2 which is fixed in later versions.

trusktr commented 9 months ago

Same breakage on Babylon v6.

like host speech animations and gaze tracking.

Also at least one character's jaw line gets deformed too (during idle no speech).

theVoiceMouse commented 9 months ago

would be great to get an upgraded / compatible version.

trusktr commented 8 months ago

@theVoiceMouse WIP :). Folks over in the Babylon forums have provided some clues as to what we need to fix:

https://forum.babylonjs.com/t/models-broken-when-updating-from-v4-to-v5/43758/2

At the moment I'm finishing up with:

and I can circle back to fixing Babylon after that. In the meantime, if you'd like to give the fix a try by branching off of #183, that'd be sweet, and I can help.

theVoiceMouse commented 8 months ago

@trusktr - amazing update. I dont do 3d modelling, so ill test your branch and upload LexV2 (working) and OpenAI support (developed, but held private incase AWS get sniffy), hopefuly you can isolate the babylon isue. Regretably board meetings next week, so will be two weeks before branch/check/commit back

Krxtopher commented 8 months ago

@theVoiceMouse it sounds like you may have already done similar work, but I did implement a solution for supporting LexV2 over in this copy of the Babylon Demos. Just never got around to contributing it back to the main repo.

theVoiceMouse commented 8 months ago

@theVoiceMouse it sounds like you may have already done similar work, but I did implement a solution for supporting LexV2 over in this copy of the Babylon Demos. Just never got around to contributing it back to the main repo.

@Krxtopher can raise the lex v2 issue andpush yours into main as a fix to it (our code is almost identical and I love how webpack5 forced us both to use pako), I’m going @trusktr ‘s non-webpack & Babylon 5 branch and I expect there will be a lot of people who can benefit from LexV2 on the webpack/main line.

trusktr commented 8 months ago

@theVoiceMouse Cool! The PR at

is now complete apart from the Babylon fix. Getting LexV2 onto that next would be sweet.

and OpenAI support

Curious about that!

We have a rough initial open-source-not-yet-licensed openai-connected not-yet-named concept here (no docs yet, but it is a Meteor.js app, and can be launched with the meteor CLI, see the start script in package.json, and also needs keys):

https://github.com/autonomys/unnamed-experiment

trusktr commented 4 months ago

Ok, I'm back at it!

Need to get this Babylon fix settled. I don't even know exactly what the issue is. @Krxtopher and @theVoiceMouse, do you know exactly what needs to be changed, and can provide some pointers?

In the meantime, I'm investigating, starting with the thread in the Babylon forum.

trusktr commented 4 months ago

The Babylon type definitions were not visible in the Babylon host source code. To help learn what's going on with Babylon more easily (and hence make a fix for Babylon after I understand it better), here's a change that enables the types: