aws-samples / aws-tools-for-babylonjs-editor

AWS Tools for Babylon.JS Editor is a suite of tools meant to interact with Babylon.JS Editor by utilizing the capabilities of AWS products.
Apache License 2.0
32 stars 9 forks source link

Feature: enable text to speech by setting a cognito pool ID property #4

Closed JuliaABurch closed 2 years ago

JuliaABurch commented 2 years ago

Notes

This change adds an inspector property (under 'Script') for a user to add a Cognito Identity Pool ID that's been set up with permissions to access Polly.

Based on the host that was added, the Polly client will be initialized with either a female or male voice. These are the only two options available in English, so I did not put much thought into representing these as a choice in the UX.

When the scene is played, the TextToSpeech feature will be instantiated.

Testing

  1. Pull this PR, follow the instructions in the package README to set up both the local plugin workspace and the editor plugin workspace. If you are using npm link, you should run that command after adding a host to the scene.
    In my personal experience, because the editor tries to npm install dependencies in the workspace (which will blow away your npm link everytime it does so), it is easier to add "@amazon-sumerian-hosts/babylon": "file:../../../local/path/to/amazon-sumerian-hosts/packages/amazon-sumerian-hosts-babylon", to the runtimeDependencies block in the config/SceneRequirements.json file in the plugin project, so that the plugin takes care of installing the dependency for you.
  2. Create a fresh workspace.
  3. "Sumerian Host Tools" -> "Add Host" -> pick a host
  4. Select the host from the scene graph on the right. Expand the 'script' node in the inspector. You may need to wait a few seconds for the editor to compile the script and display the custom attribute.
  5. Add a Cognito Identity Pool ID here.
  6. Press play. You shouldn't see any errors - so long as the Cognito Identity Pool ID is set up correctly.
  7. Save and reload. You'll see that the Cognito Identity Pool ID is serialized and deserialized.

For a customer to use this feature, I imagine they may add something like the following block of code to the onInitialize() block of the sumerianhost.ts file in their project workspace:

    this._scene.onKeyboardObservable.add((kbInfo) => {
      if (
        kbInfo.type === KeyboardEventTypes.KEYDOWN &&
        kbInfo.event.code === 'Space'
      ) {
        console.log('Spacebar was pressed');
        this.speak('Spacebar was pressed.');
      }
   });

If they wish to add this functionality to a different script/object, they can instead interact with the SumerianHost object like this:

// change name to whatever the name of the added host was
const host = scene.getNodeByName('Christine') as SumerianHost;
host.speak("Hello world!");

See this BabylonJS Editor issue for other ways the user may get a reference to the SumerianHost object.

Additionally, they will need to add this to the module.exports.resolve block in the webpack.config.js for their project:

            modules: ['node_modules'],
            alias: {
                '@babylonjs/core': path.resolve('./node_modules/@babylonjs/core')
            }

Without this, you may see errors when the scene loads like:

_babylonjs_core_Engines_engine__WEBPACK_IMPORTED_MODULE_2__.Engine.audioEngine is undefined

_setAudioContext TextToSpeechFeature.js:44

Known Issues

  1. "Fiona" is not known to play well with runtime behaviors; avoid choosing this host
  2. The label in the inspector that displays "Cognito Identity Pool ID" wraps oddly (unless the inspector is expanded.) There's no nice tooltip functionality, and it might be confusing just to shorten it to "Cognito Id".
  3. I would like to move more of the typing information defined here into @amazon-sumerian-hosts/babylon proper. This is in progress.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.