search

jfversluis / Plugin.Maui.Audio

Plugin.Maui.Audio provides the ability to play audio inside a .NET MAUI application
MIT License
259 stars 45 forks source link

Sending Linear PCM audio chunks to a cloud service that has whisper deployment for transcription in an efficient way near real time #114

Open radrad opened 5 months ago

radrad commented 5 months ago

If we use Linear PCM (LPCM), which doesn't include headers and allows any segment of the audio to be played independently, we can send the audio chunks to the backend. With a method for silence detection, we could split the real-time audio into chunks and send these to OpenAI's Whisper for transcription. This approach could enable near real-time transcription display on the frontend, store the transcription, and perform summarization or other AI tasks.

I'm interested in finding out how to send the minimum amount of bytes with sufficient quality to a cloud-based Whisper deployment for transcription. This transcription could be saved as metadata for the audio recording, and the audio itself backed up to a cloud location. Additionally, generating .srt files with timestamps would allow users to jump to specific audio segments corresponding to the subtitles.

jfversluis commented 2 months ago

Hey there, sorry we didn't get to this earlier. I'm... Not entirely sure what you're asking here? :)