Recording Audio/Video in a real time call

[prabhakarreddy1234]

Is it possible to record Audio and/or Video in a real time call. I am using Skype for Business Channel and creating a group conference. Bot will be part of group conference and it records audio/video and later transcribes that media using Azure Cognitive services. Is this use case supported by current version of RealTimeMediaCalling SDK ?

[MalarGit]

Real time media support is not available for Skype for business. Also please note that group conference in Skype has also been disabled for bots due to privacy reasons.

[annevah]

Can we do this for Skype? Though it is intended for video there is no sample for recording the video/audio of the user. Any help on how to do this.

[ssulzer]

@annevah

In a 1:1 Skype video call with your real-time media bot, the bot has access to the Skype caller's audio and video. The Huebot sample shows a bot receiving the audio and video of the Skype caller. To create a recording, the bot could save the contents of audio and video media buffers to disk and use a tool like ffmpeg to process them into an MP4 file, for example. But there's no specific recording sample. Also note that the bot cannot control what resolution it receives of the Skype caller's video, and will likely get a low resolution such as 240p.

[samigranfors]

@ssulzer

How is the resolution determined? We are currently getting 180p and 240p in our tests with the Huebot. Can we get a better resolution?

[ssulzer]

@samigranfors

Currently there is no API to request a different video resolution from the peer in a 1:1 call, so you will receive a low resolution like 180p or 240p by default.

We are planning to enable this soon, by introducing some new APIs on the VideoSocket class. I expect an update of the Microsoft.Skype.Bots.Media nuget package by end of June will provide the ability to request a higher resolution. Thanks.

[annevah]

@ssulzer could you please guide me on how I can develop an IVR type bot that receives the video of the user ? I have an IVR bot up and running, but as I said I want to receive the video of the user. If I am using real time media player, how do we determine on when to ask the next question or repeat the question etc.

[vinaychandra]

@annevah, IVR bots by design cannot receive media streams. You should use a RTM bot in order to receive any media stream.

[annevah]

@vinaychandra I know that. I am asking how do we develop a IVR type bot using real time media i.e how to determine when to ask the next question/ when the user has answered the question etc. using real time media.

[ssulzer]

@annevah

To build a video-capable IVR bot, especially one which can see the caller in real-time, requires the use of the real-time Microsoft.Skype.Bots.Media APIs and not the "higher-level" PlayPrompt/Recognize IVR workflow APIs (which do not provide support for video). Building such a bot on the real-time media platform means building a custom IVR workflow engine. That will involve:

• maintaining a dialog state machine to keep track of where the bot is in the conversation graph and whether it is playing a prompt or listening for a response
• loading audio and video files into memory and playing them with the AudioVideoFramePlayer (see the AudioVideoPlayerBot sample)
• for a speech-enabled bot, using an API like the Microsoft Speech APIs to perform continuous speech recognition on incoming audio (received via an AudioSocket) and parsing the recognition result (see the HueBot sample for an example)
• for listening to dial-pad digits entered by the caller, use the AudioSocket's ToneReceived event
• for detecting whether the caller is speaking or not, use either the speech recognition APIs, or check the IsSilence flag on the received AudioMediaBuffers

[annevah]

@ssulzer Thank you so much for that detailed explanation.

[cioxideru]

@ssulzer I can start working with my realtime-media-bot without ssl certificate? as well as how can it be deployed on a local machine for debugging, with self signed certificate? thank you

[vinaychandra]

@cioxideru Local debugging with RealTimeMedia is asked for, multiple times and it is in our backlog. The current library does not support local debugging and can only run on instances which can be reachable from the public internet. Along with that you always require a valid certificate from a trusted Certificate Authority (CA) and self-signed certificates are not accepted.

[cioxideru]

@vinaychandra I can run it on my hosting(windows server 2012 r2), with dns and a valid certificate for it? So I create a Skype channel and point to this dns? or the requirement of hosting in Azure is mandatory? Many thanks for your help!!!!!

[vinaychandra]

@cioxideru Please take a look at this doc that outlines on how the media library can be setup. You can run it on a Windows Server machine if you are able to satisfy all the prerequisites.

[ssulzer]

Hello @samigranfors

There is now an API on the VideoSocket to request a higher resolution from the Skype caller, available in the recently released version 1.9 of the Microsoft.Skype.Bots.Media nuget package [https://www.nuget.org/packages/Microsoft.Skype.Bots.Media/1.9.0.1-alpha].

In a 1:1 Skype<->bot video call, use the VideoSocket.Subscribe(VideoResolution preferredVideoResolution) method to specify what resolution your bot wishes to receive from the caller. Note, the bot may not receive the exact resolution requested.

[suchism]

i am facing issues running HueBot sample locally with ngrok . when i hit the messages api from Test Web Client i get below requests on ngrok utility. POST /api/messages 404 Not Found OPTIONS /api/messages 200 OK Any pointers , where i am going wrong ?

[bijaykush]

@ssulzer @ksikorsk I am looking for a process to record the buffer data is being received on

private void OnAudioMediaReceived(object sender, AudioMediaReceivedEventArgs e)
        {
            this.GraphLogger.Info($"Received Audio: [VideoMediaReceivedEventArgs(Data=<{e.Buffer.Data.ToString()}>, Length={e.Buffer.Length}, Timestamp={e.Buffer.Timestamp})]");
                      // TBD: Compliance Recording bots can record the Audio here
            e.Buffer.Dispose();
        }

This code snippet is from Compliance Recording Bot sample https://github.com/microsoftgraph/microsoft-graph-comms-samples/blob/master/Samples/BetaSamples/LocalMediaSamples/ComplianceRecordingBot/FrontEnd/Bot/BotMediaStream.cs

As per your instruction on the above comments (save audio buffer data to disk and use tools like ffmpeg), I tried to convert the incoming handler of data to the byte array and save to a local disk, but since the function is being triggered continuously on the socket as recording/call continues. I am being confused.

I am looking for any possible way or sample or pointer to record this call in compliance recording bot.

[Mishra-Saurabh]

Intro: We are creating a bot to record the team meeting. After the meeting gets over we should get the recordings files in video format.

Issue: We have implemented this code(https://github.com/microsoftgraph/microsoft-graph-comms-samples/tree/master/Samples/V1.0Samples/AksSamples/teams-recording-bot) using Azure Kubernetes. We can join the team meeting and recording start automatically and we get audio wav file format in the zip file to our local but we need video files. So how can we get the video files in a .zip file to our local?

Implementation of code: We have used a video socket. But we don't know how we can implement code to get video so that we can save that video to our local .zip file. Please help us with this.

`private void OnVideoMediaReceived(object sender, VideoMediaReceivedEventArgs e) { this.GraphLogger.Info($"[{e.SocketId}]: Received Video: [VideoMediaReceivedEventArgs(Data=<{e.Buffer.Data.ToString()}>, Length={e.Buffer.Length}, Timestamp={e.Buffer.Timestamp}, Width={e.Buffer.VideoFormat.Width}, Height={e.Buffer.VideoFormat.Height}, ColorFormat={e.Buffer.VideoFormat.VideoColorFormat}, FrameRate={e.Buffer.VideoFormat.FrameRate})]");

        // TBD: Compliance Recording bots can record the Video here
        e.Buffer.Dispose();
    }`