search

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
22.15k stars 3.31k forks source link

.Net: Add AssemblyAI connector #5392

Open Swimburger opened 8 months ago

Swimburger commented 8 months ago

Motivation and Context

AssemblyAI is a speech AI company offering AI models through APIs. Adding a connector will help users integrate AssemblyAI easily with Semantic Kernel.

Description

Progress of implementation of AssemblyAI connector. Current implementation ASSEMBLYAI BRANCH

TODO

  1. AudioToTextService
    • [x] GetTextContentsAsync using AudioContent (#5094)
    • [x] GetTextContentsAsync using AudioStreamContent (#5094) (deprecated in favor of file service)
    • [x] Add DI extensions (#5094)
    • [x] Add AssemblyAI file service to upload files (#5964)
    • [x] Return typed class in TextContent.InnerContent
    • [x] Add all transcript parameters to AssemblyAIAudioToTextExecutionSettings

Potential additions

Swimburger commented 8 months ago

I noticed that the IAudioToTextService.GetTextContentsAsync method returns multiple TextContent's. We have APIs to return the transcript as sentences and another as paragraphs. Would it make sense to add options to AssemblyAIAudioToTextExecutionSettings, which would control whether the transcript is returned as a single TextContent, or a TextContent for each sentence, or a TextContent for each paragraph?

Krzysztof318 commented 8 months ago

I would add to todo also full realtime transcribing, so you send AudioContent or AudioStreamContent and you get IAsyncEnumerable<StreamingTextContent>

Swimburger commented 8 months ago

I would add to todo also full realtime transcribing, so you send AudioContent or AudioStreamContent and you get IAsyncEnumerable<StreamingTextContent>

I want to add realtime, but I want to finalize and release non-realtime transcription first.

Our realtime solution uses a WebSocket connection, expects raw audio bytes to be sent continuously, and responds with partial and final transcript objects. This is mostly consistent with other realtime transcription services. I'd be happy to work with y'all in figuring out how to create a good abstraction that'll work for us and other realtime services.

Swimburger commented 7 months ago

Instead of using the AudioStreamContent, I'm introducing an AssemblyAI file service for users to upload their files to AssemblyAI. #5964

In the future, we can use a streaming audio content class for Streaming STT.

Swimburger commented 5 months ago

Now that we have the AssemblyAIAudioToTextService and AssemblyAIFileService in, I think we can release the initial version of this connector. What would the next steps be?

Swimburger commented 1 month ago

This PR uses the AssemblyAI SDK: https://github.com/microsoft/semantic-kernel/pull/8556

Swimburger commented 3 weeks ago

@RogerBarreto With the SDK PR merged, is it ready to be released?

Swimburger commented 1 week ago

Ping!

Krzysztof318 commented 1 week ago

Pong!

Catch up with Roger via discord, maybe he knows when team could review your code.

You should also take a look at new abstraction microsoft.extensions.ai and introduce it directly in your sdk assemblyAi. This will be a new way of implementing connectors.