This is a large PR. Fundamentally, it introduces the first version of a voice interaction feature. It should be working pretty well however there is not full test coverage so the feature is initially disabled. It can be enabled in features.yml
The core functionality is: turn on the mic, speak aloud instead of typing, responses are spoken back, any time you say "see this" or "my screen" or other references which indicate you're looking at something it will take a screenshot, if you say "hold on Samantha" then it will mostly stop listening but you can reinvoke it by saying something like "Samantha, I'm back." Here is a short demo video:
Moving all stimulus controllers into javascript/stimulus subdirectory
Introduction of a new pattern called blocks. The core classes are called interfaces. These live in javascript/blocks. Stimulus controllers interact with interfaces (e.g. Microphone, Transcriber, Listener, Speaker).
There are also blocks service classes which wrap browser controls to the microphone, speaker, and screen viewing.
To improve the javascript development experience, blocks utilities are loaded from blocks/lib. A bunch of ActiveSupport style functions have been introduced on core javascript models like array and string.
The plumbing for blocks is built around node & jest, but node is not required to run the app. It's only for a nicer developer experience.
The heart of blocks is a base class called ReadableClass which creates a much simpler syntax for any javascript class (it's a bit like ActiveModel).
Fixed a bug with auto-scrolling down after images are loaded into messages.
There is a fair amount of plumbing for blocks:
bin/blocks console and bin/blocks test are two utilities for
All of the blocks modules are loaded in the browser (via application.js and blocks/index.js), in blocks console (via bin/blocks), and in blocks jest test environment (via test_helper.js)
All the blocks libraries load before stimulus so that stimulus controllers can reference blocks. This exposed a slight race condition in the app which was discovered via system tests. That's why there is some small reworking of system tests, it was to fix this bug by having a couple stimulus controllers interact via outlets rather than events.
This is a large PR. Fundamentally, it introduces the first version of a voice interaction feature. It should be working pretty well however there is not full test coverage so the feature is initially disabled. It can be enabled in features.yml
The core functionality is: turn on the mic, speak aloud instead of typing, responses are spoken back, any time you say "see this" or "my screen" or other references which indicate you're looking at something it will take a screenshot, if you say "hold on Samantha" then it will mostly stop listening but you can reinvoke it by saying something like "Samantha, I'm back." Here is a short demo video:
https://share.zight.com/L1ubg6j4
Included in this PR:
interfaces. These live in javascript/blocks. Stimulus controllers interact with interfaces (e.g. Microphone, Transcriber, Listener, Speaker).There is a fair amount of plumbing for blocks:
bin/blocks consoleandbin/blocks testare two utilities for