I upgraded Webpack to v5 in these PR #2267 due to the its improved asset handling and over it I have created a functional POC that I will publish as a WIP PR so you can take a look to the code added.
The highlight design:
I have a class AudioMain, this class is responsable for:
Initialize a webworker (AudioWorker) that holds the persistent websocket connection.
Setup AudioWorkets and transfer its MessagePort to the AudioWorker.
Notify UI about connection state, source state and sink state.
Enable client keyword spotting functionality and propagate the detection to the worker.
Then the AudioWorker is in charge of:
Create a persistent connection to the audio-pcm websocket.
Instruct AudioMain to setup the source or sinks and transfer their MessagePort to the AudioWorker.
Transfer the audio from the WebSocket to the appropriate MessagePort or viceversa, doing reencode and resample to the audio data as needed.
Propagate to AudioMain the connection state changes.
I need a lot of feedback.
Initially I have placed the mentioned code under src/js/voice and I have added the options to the src/components/theme-switcher.vue. Also I have created a src/components/dialog-mixin.js for the src/components/app.js and from there the initialization is done when the option is enabled.
I will like to add some kind of icon that inform the user about the recording state and also that can be used to trigger the dialog by clicking on it, but I don't know where to place it or what kind of design I should use.
I will open the WIP PR when I have a moment and mention these issue because I think it will be easier to comment over what it's done.
GiviMAD
commented
6 months ago
Hello,
I want to add to the UI a client for the websocket protocol added by these PR https://github.com/openhab/openhab-core/pull/4032 to use the voice features from the UI.
I upgraded Webpack to v5 in these PR #2267 due to the its improved asset handling and over it I have created a functional POC that I will publish as a WIP PR so you can take a look to the code added.
The highlight design:
I have a class AudioMain, this class is responsable for:
Enable client keyword spotting functionality and propagate the detection to the worker.
Then the AudioWorker is in charge of:
I need a lot of feedback.
Initially I have placed the mentioned code under
src/js/voiceand I have added the options to thesrc/components/theme-switcher.vue. Also I have created asrc/components/dialog-mixin.jsfor thesrc/components/app.jsand from there the initialization is done when the option is enabled.I will like to add some kind of icon that inform the user about the recording state and also that can be used to trigger the dialog by clicking on it, but I don't know where to place it or what kind of design I should use.
I will open the WIP PR when I have a moment and mention these issue because I think it will be easier to comment over what it's done.
Best Regards