Migrate firmware to stream from PCM to OPUS

josancamon19 commented 1 week ago

Is your feature request related to a problem? Please describe. Better audio quality, less bandwitdh.

Describe the solution you'd like Firmware should stream audio bytes using Opus codec, and the mobile app should receive those bytes properly modifying respectively on transcript.dart to handle that, do the .

Describe alternatives you've considered uLaw vs Opus, I don't know which one is better, from what I've been suggested Opus, but lmk if you think differently.

diegoasua commented 1 week ago

The other way. PCM to Opus. Can't change title

ebariaux commented 3 days ago

Last update on my current status I posted on Discord around that topic:

I have µ-Law encoding working with Friend and my own app
I created a PR for the change on firmware side ONLY (https://github.com/BasedHardware/Friend/pull/356), which is straight forward
This bring 16kHz (meaning frequency range up to 8kHz) with the same bandwidth
I did not update the Friend app and do not intend to (I'm not a Flutter/Dart dev and not intersted in trying besides trivial changes). With a quick look I see that some work in wav_bytes.dart would be required (e.g. WAV header that hardcodes PCM) and certainly other places (also depending on what you want to send to the SST service)
I did publish my native iOS code in https://github.com/nelcea/PAL. If you look into https://github.com/nelcea/PAL/blob/main/iOS/PALApp/Codecs.swift, in the µLawCodec class you'll see the simple logic I use to decode the signal to PCM using a static lookup table. This might be an inspiration for whomever wants to have a go at the change in the Friend app
I'm still work on the Opus codec but getting some errors during the decoding on iOS. I need to go deeper into the original opus lib doc/code.
The size gain I saw between Opus over µ-law is quite small (10 %, unless the encoding is wrong). So my current thinking is that given I believe the audio quality would be only slightly better with Opus but the processing load for encoding/decoding is greater, I would go for µ-law at this stage.
As a side note, it seems the device could go slightly higher to a bit more than 20kHz sampling frequency, given us an additional 2kHz for the frequency range

ebariaux commented 3 days ago

And in case somebody wants to test with Opus, changes required in the firmware are:

Enabled CODEC_OPUS in config.h
Add CONFIG_CODEC_OPUS=y in prj.conf

BasedHardware / Friend

Migrate firmware to stream from PCM to OPUS #317