Got test_mocks.js to load in the browser for rails system tests. This enables activating voice mode with a stubbed out microphone and other services so I can write full integration tests. It involved some refactoring of blocks.
Completely rewrote the tricky speaker_controller.js in order to fix a number of subtle bugs that were leading to an unreliable voice experience + it enables the playback of individual messages with a play button beneath them (w/o enabling voice mode). Notably, instead of listening for various Turbo JS events it now uses MutationObserver which is a more fundamental solution.
Major functional changes:
Do not keep listening after SpeakTo called (if a user keeps talking it would submit another SpeakTo while it's still processing the first one)
If the Listener is Dismissed and the AI initiates conversation (e.g. because of a reminder) then it will speak and Invoke Listener so you can speak
Added support for manually clicking to have a voice speak a message
Add support for the assistant replying twice in a row without a user message in between
Updated the blocks test suite so that the few tests that exist there are all passing again.
Add thorough system tests for the playback & speaker controllers in the browser
Fixed a race condition inherent in ActionCable / Turbo Stream where an updated message can broadcast in the split second between when the message is being displayed on screen and when the turbo stream connection is established to listen for broadcasts.
Corrected the pop sound occurring even before you had approved audio
Added an alert to explain why the feature isn't working if you have microphone disabled
Fixed a few flaky system tests and reorganized the system test files a bit
Known issues that remain:
Investigate the ' vs '' thing that is streaming in
On my pixel tablet, the keyboard is auto-popping up when the form submits
Text to speech for F and C is not correct
If you talk for a long time (or start talking right before the auto-dismiss is going to happen) then the listener dismiss timeout will fire even while you are still speaking. We need to figure out how to detect speech even before the person has finished sharing their thoughts. I used to do this by enabling a Microphone at the same time as the Transcriber, but that is not supported by mobile browsers and was creating bugs.
Investigate whether a PWA app can prevent the device from going to sleep because the screen turns off while using the app with your voice.
On mobile, when the screen goes to sleep and you reactivate it, the microphone stops working and prevents you from speaking to it.
I've addressed most of the issues of "thinking sound goes forever" because they were front end bugs where newly streamed in text was not being detected properly. However, I think it's still possible to have an API issue that could cause the front-end to hang and be "thinking forever." Once I confirm that this happens, I probably need some kind of front-end timeout that fires after X seconds of "thinking" with no response so that the user can at least speak again to re-initiate hands free.
The foundational work in this PR:
test_mocks.jsto load in the browser for rails system tests. This enables activating voice mode with a stubbed out microphone and other services so I can write full integration tests. It involved some refactoring of blocks.speaker_controller.jsin order to fix a number of subtle bugs that were leading to an unreliable voice experience + it enables the playback of individual messages with a play button beneath them (w/o enabling voice mode). Notably, instead of listening for various Turbo JS events it now uses MutationObserver which is a more fundamental solution.Major functional changes:
Known issues that remain: