search

sipsorcery-org / sipsorcery

A WebRTC, SIP and VoIP library for C# and .NET. Designed for real-time communications apps.
https://sipsorcery-org.github.io/sipsorcery
Other
1.44k stars 438 forks source link

GetStartedVideo VP8 decode attempt failed, Bitstream not supported by this decoder error #344

Closed thornzz closed 3 years ago

thornzz commented 3 years ago

Hi,

I have started GetStartedVideo example. Callee side can see my video but i cant see callee side's video form that after run project its grey and at console im getting "VP8 decode attempt failed, Bitstream not supported by this decoder" error.

Why i cant see video target's stream?

sipsorcery commented 3 years ago

Why i cant see video target's stream?

Assumedly because the VP8 video codec cannot decode the video signal from the RTP packets.

thornzz commented 3 years ago

There are 2 other video codecs our softphone client's supports that H263 and H263-1998. When i close VP8 and only open H263 codecs from other side video calls not starting and there is only audio call. So what i have to ?

sipsorcery commented 3 years ago

The only video codecs I've implement are VP8 in SIPSorceryMedia.Encoders and H264 in SIPSorcery.FFmpeg.

FFmpeg does support H263, along with most other codecs, but I don't have any plans to wire up H263. It would also need the RTP packetisation logic.

thornzz commented 3 years ago

So how can i use SIPSorceryMedia.Encoders and H264 in SIPSorcery.FFmpeg ? Is there any example for them?

sipsorcery commented 3 years ago

The WebRTC Examples that use video will use one or both those libraries.

thornzz commented 3 years ago

I check the codes but I'm a little confused. Can you give me a little example for implementing SIPSorceryMedia.Encoders to Windows Softphone?

sipsorcery commented 3 years ago

The way it works is:

The SoftPhone example is already using SIPSorceryMedia.Windows and it in turn is using SIPSorceryMedia.Encoders for encoding.

thornzz commented 3 years ago

Thanks i will check it out.

thornzz commented 3 years ago

I have tried with Softphone Windows example but no success. When i try to call another number its ringing than callee is accepting it with a Video Call from. Audio call is starting but video is not working. I have tried another test that other client call SoftPhone Example client and accept it then send Video call request its giving some errors. Could you check them?

This is Softphone Example client is calling other client. Getting this error => System.ApplicationException: 'The Windows Video End Point does not support external samples. Use the video end point from SIPSorceryMedia.Encoders.'

This is when other client call softphone example client and other client req video call.

[14:42:03 DBG] Request Received udp:0.0.0.0:5060<-udp:10.0.35.10:5060: BYE sip:25@10.0.35.138:5060 SIP/2.0. [14:42:04 DBG] Request Received udp:0.0.0.0:5060<-udp:10.0.35.10:5060: INVITE sip:10.0.35.138:5060 SIP/2.0. [14:42:04 DBG] Re-INVITE request received INVITE sip:10.0.35.138:5060 SIP/2.0. [14:42:04 DBG] Replacing existing remote audio track for ssrc 1606427680. [14:42:04 DBG] Replacing existing remote video track for ssrc 0. [14:42:04 DBG] Setting audio sink and source format to 0:PCMU 8000. [14:42:04 INF] Incoming call request: udp:0.0.0.0:5060<-udp:10.0.35.10:5060, uri:sip:10.0.35.138:5060. [14:42:04 DBG] Response Sent udp:10.0.35.138:5060<-udp:10.0.35.10:5060: INVITE 200 Ok. [14:42:04 DBG] Set remote audio track SSRC to 1606427680. [14:42:05 DBG] Request Received udp:0.0.0.0:5060<-udp:10.0.35.10:5060: ACK sip:10.0.35.138:5060 SIP/2.0. [14:42:05 DBG] RTCP BYE received for SSRC 1606427680, reason . [14:42:05 DBG] RTCP session removing reception report for remote ssrc 1606427680. [14:42:05 DBG] RTCP BYE received for SSRC 1606427680, reason . [14:42:05 DBG] Set remote audio track SSRC to 1606427680. [14:42:05 WRN] Could not match an RTCP packet against any SSRC's in the session. [14:42:07 DBG] Request Received udp:0.0.0.0:5060<-udp:10.0.35.10:5060: BYE sip:10.0.35.138:5060 SIP/2.0. [14:42:07 INF] Remote call party hungup BYE sip:10.0.35.138:5060 SIP/2.0. [14:42:07 DBG] Response Sent udp:10.0.35.138:5060<-udp:10.0.35.10:5060: BYE 200 Ok. [14:42:07 DBG] RTPChannel closing, RTP receiver on port 57778, Control receiver on port 57779. Reason: normal. [14:42:07 DBG] RTPChannel closing, RTP receiver on port 57780, Control receiver on port 57781. Reason: normal.

sipsorcery commented 3 years ago

System.ApplicationException: 'The Windows Video End Point does not support external samples. Use the video end point from SIPSorceryMedia.Encoders.'

That's a bug. If the softphone demo can't access a webcam it should fallback to a static test pattern video source. That's currently not wired up correctly due to some recent changes. I'll fix it shortly.

If the softphone demo is able to access a webcam you won't get that exception. Here's a screenshot of the softphone demo in a video call with MicroSIP on my Windows 10 PC.

softphone-to-microsip

microsip_videosettings

thornzz commented 3 years ago

Thanks. I have checked again when Softphone example client call other client there is no problem with video. But when other client call Softphone example and req video call after call is started its getting this error.. In our other client its not possible to start call with video call. Video call is starting after call started and req video call. So I think i need to modify code but need help

[11:53:24 DBG] Request Received udp:0.0.0.0:5060<-udp:10.0.35.10:5060: INVITE sip:10.0.35.138:5060 SIP/2.0. [11:53:24 DBG] Re-INVITE request received INVITE sip:10.0.35.138:5060 SIP/2.0. [11:53:25 DBG] Replacing existing remote audio track for ssrc 1341192132. [11:53:25 DBG] Replacing existing remote video track for ssrc 0. [11:53:25 DBG] Setting audio sink and source format to 0:PCMU 8000. [11:53:25 INF] Incoming call request: udp:0.0.0.0:5060<-udp:10.0.35.10:5060, uri:sip:10.0.35.138:5060. [11:53:25 DBG] Response Sent udp:10.0.35.138:5060<-udp:10.0.35.10:5060: INVITE 200 Ok. [11:53:25 DBG] Set remote audio track SSRC to 1341192132. [11:53:25 DBG] Request Received udp:0.0.0.0:5060<-udp:10.0.35.10:5060: ACK sip:10.0.35.138:5060 SIP/2.0. [11:53:25 DBG] RTCP BYE received for SSRC 1341192132, reason . [11:53:25 DBG] RTCP session removing reception report for remote ssrc 1341192132. [11:53:25 DBG] Set remote audio track SSRC to 1341192132. [11:53:25 DBG] RTCP BYE received for SSRC 1341192132, reason . [11:53:25 DBG] RTCP session removing reception report for remote ssrc 1341192132. [11:53:25 DBG] Set remote audio track SSRC to 1341192132. [11:53:25 WRN] Could not match an RTCP packet against any SSRC's in the session.

sipsorcery commented 3 years ago

Make sure you click the video call button on the softphone you are placing the call from. The default call button on most softphones typically initiates an audio only call.

The sipsorcery softphone demo only answers with video if the incoming call request offered video.

thornzz commented 3 years ago

Yes i see it. But Our softphone is 3cx pbx server's softphone client. Its only got one Call button that making call is audio. But after call is connected you can click video button than its sending re-invite with video req packets. So i have to handle this situation. When softphone example got re-invite with video req packets i have to start video call with each participant. How can i achieve this? Is it possible?

sipsorcery commented 3 years ago

It's possible to add video to an existing call in response to a re-invite request. It hasn't come up before so there's no logic for it. If you wanted to add it the place to start is here.

thornzz commented 3 years ago

It's possible to add video to an existing call in response to a re-invite request. It hasn't come up before so there's no logic for it. If you wanted to add it the place to start is here.

I could not get working re-invite request logic. :( So i have to wait someone code re-invite request logic. I will use SipSorcery Softphone client for all participants. But i got another question. I have developed a project my own sipphone client with Windows Forms.

I have implemented this code. But i can only see remote side video. How can i get Local video?

private void SIPCallAnswered(SIPClient client) { if (client == _sipClients[0]) { if (_sipClients[1].IsCallActive && !_sipClients[1].IsOnHold) { //_sipClients[1].PutOnHold(_onHoldAudioScopeGL); _sipClients[1].PutOnHold(); }

            BeginInvoke((Action)delegate ()
            {
                btnCevapla.Enabled = false;
                btnReddet.Enabled = false;
                btnIptal.Enabled = false;
                btnSonlandir.Enabled = true;
                btnBeklet.Enabled = true;
                btnHatAl.Enabled = true;
                btnBeklet.Enabled = true;
                btnKorTransfer.Enabled = true;
                if (_sipClients[0].MediaSession.HasVideo)
                {
                    _sipClients[0].MediaSession.OnVideoSinkSample += (sample, width, height, stride, pixelFormat) =>

                         localVideo.BeginInvoke(new Action(() =>
                         {
                             unsafe
                             {
                                 fixed (byte* s = sample)
                                 {
                                     System.Drawing.Bitmap bmpImage = new System.Drawing.Bitmap((int)width, (int)height, stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, (IntPtr)s);
                                     localVideo.Image = bmpImage;
                                 }
                             }
                         }));
                }
            });
        }
        else if (client == _sipClients[1])
        {
            BeginInvoke((Action)delegate ()
            {
                btnCevapla.Enabled = false;
                btnReddet.Enabled = false;
                btnIptal.Enabled = false;
                btnSonlandir.Enabled = true;
                btnBeklet.Enabled = true;
                btnTransfer.Enabled = true;

                if (_sipClients[1].MediaSession.HasVideo)
                {
                    _sipClients[1].MediaSession.OnVideoSinkSample += (sample, width, height, stride, pixelFormat) =>

                        localVideo.BeginInvoke(new Action(() =>
                        {
                            unsafe
                            {
                                fixed (byte* s = sample)
                                {
                                    System.Drawing.Bitmap bmpImage = new System.Drawing.Bitmap((int)width, (int)height, stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, (IntPtr)s);
                                    localVideo.Image = bmpImage;
                                }
                            }
                        }));
                }
            });

            if (_sipClients[0].IsCallActive)
            {
                if (!_sipClients[0].IsOnHold)
                {
                    //_sipClients[0].PutOnHold(_onHoldAudioScopeGL);
                    _sipClients[0].PutOnHold();
                }
            }
        }
    }
sipsorcery commented 3 years ago

I have implemented this code. But i can only see remote side video. How can i get Local video?

The best option is the MediaCapture API.

thornzz commented 3 years ago

I have used Aforge.net library for local video capture but when video call starting im getting "The Windows Video End Point does not support external samples. Use the video end point from SIPSorceryMedia.Encoders." error because of webcam is using by Aforge.net and Sipsorcery cant reach webcam. What i have to do ?

sipsorcery commented 3 years ago

Use the VideoEncoderEndPoint from SIPSorceryMedia.Encoders instead of the WindowsVideoEndPoint. They both implement the same interface so you can swap one for the other.

Although reading back over your original issue if you are still unable to use VP8 there's no point using the VideoEncoderEndPoint as that's the only format it supports. Instead you can swap the WindowsVideoEndPoint for SIPSorceryMedia.FFmpeg.VideoEndPoint from SIPSorceryMedia.FFmpeg. It supports VP8 and H264.

thornzz commented 3 years ago

VP8 is enough for me . I have changed WindowsVideoEndpoint to VideoEncoderEndpoint but this time there is no video call request to remote side so i can only use audioCall. Could you check any problem with code?

private VoIPMediaSession CreateMediaSession() { var windowsAudioEndPoint = new WindowsAudioEndPoint(new AudioEncoder(), m_audioOutDeviceIndex); var VideoEndPoint = new VideoEncoderEndPoint();

    MediaEndPoints mediaEndPoints = new MediaEndPoints
    {
        AudioSink = windowsAudioEndPoint,
        AudioSource = windowsAudioEndPoint,
        VideoSink = VideoEndPoint,
        VideoSource = VideoEndPoint,
    };

    var voipMediaSession = new VoIPMediaSession(mediaEndPoints);
    voipMediaSession.AcceptRtpFromAny = true;

    return voipMediaSession;
}
sipsorcery commented 3 years ago

If VP8 is now sufficient why can't you use the GetStartedVideo demo as your starting point?? We seem to be going around in circles here.

var windowsAudio = new WindowsAudioEndPoint(new AudioEncoder());
var videoEndPoint = new SIPSorceryMedia.Encoders.VideoEncoderEndPoint();
MediaEndPoints mediaEndPoints = new MediaEndPoints
{
    AudioSink = windowsAudio,
    AudioSource = windowsAudio,
    VideoSink = videoEndPoint,
    VideoSource = videoEndPoint,
};
var voipMediaSession = new VoIPMediaSession(mediaEndPoints);
voipMediaSession.AcceptRtpFromAny = true;

var offer = voipMediaSession.CreateOffer(null);
Console.WriteLine(offer);

Which generates an SDP offer with video.

v=0
o=- 1521050359 0 IN IP4 127.0.0.1
s=-
c=IN IP4 192.168.0.50
t=0 0
m=audio 62206 RTP/AVP 0 8 9 101
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:9 G722/16000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=sendrecv
m=video 62208 RTP/AVP 96
a=rtpmap:96 VP8/90000
a=sendrecv
thornzz commented 3 years ago

Hi again,

I think i could not explain the problem. When i use WindowsVideoEndpoint in our client that created from SipSorcerySoftPhone example codes each participant can get remote videos. There is no problem. But this is not enough. I want to show local video besides remote video. When i use Aforge.Net library for MediaCapture with WindowsVideoEndpoint class this time i can see Local video but Sipsorcery getting for remote video "The Windows Video End Point does not support external samples. Use the video end point from SIPSorceryMedia.Encoders. " error.

When i use SIPSorceryMedia.Encoders for VideoEndpoint (For testing commented Local video codes) this time Remote Video not working. When you use SIPSorceryMedia.Encoders for Softphone example you will see what i mean.

Im getting video frames and showing at PictureBox control with this code.

                 if (_sipClients[0].MediaSession.HasVideo)
                          {
                    _sipClients[0].MediaSession.OnVideoSinkSample += (sample, width, height, stride, pixelFormat) =>
                        //Remote Video => PictureBox Control
                         remoteVideo.BeginInvoke(new Action(() =>
                         {
                             unsafe
                             {
                                 fixed (byte* s = sample)
                                 {
                                     System.Drawing.Bitmap bmpImage = new System.Drawing.Bitmap((int)width, (int)height, stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, (IntPtr)s);
                                     remoteVideo.Image = bmpImage;
                                 }
                             }
                         }));
                }

Here is the screenshot remote side working good with WindowsVideoEndPoint..

Screenshot : https://prnt.sc/vijir1

My main goal is make SipSorcery.Softphone example with Local and Remote Video. Example is only got Remote Side video with WindowsVideoEndpoint class.

sipsorcery commented 3 years ago

My main goal is make SipSorcery.Softphone example with Local and Remote Video. Example is only got Remote Side video with WindowsVideoEndpoint class.

It would only require a small change to the WindowsVideoEndPoint.FrameArrivedHandler to add a new event that supplies the local video samples.

The easiest approach is to supply a SoftwareBitmap which you'll most likely need to convert to a normal Bitmap but there are lots of examples for that.

thornzz commented 3 years ago

It would only require a small change to the WindowsVideoEndPoint.FrameArrivedHandler to add a new event that supplies the local video samples.

I have added an event to SipSorcery.Windows project named OnLocalVideoEncodedSample. And then invoke this event inside of FrameArrivedHandler Event. For invoke parameter i get the SoftwareBitmap object and send to OnLocalVideoEncodedSample event.

Everything working good but if i interact with client form im getting "System.ObjectDisposedException: 'The object has been closed.". I have commented dispose methods but it does not work. Here is my code could you give me idea to fix it?

               private async void FrameArrivedHandler(MediaFrameReader sender, MediaFrameArrivedEventArgs e)
                     {
                      if (!_isClosed)
                            {
                                   if (!_isClosed && OnVideoSourceEncodedSample != null)
                                     {
                                      using (var mediaFrameReference = sender.TryAcquireLatestFrame())
                                         {
                                           var videoMediaFrame = mediaFrameReference?.VideoMediaFrame;
                                           var softwareBitmap = videoMediaFrame?.SoftwareBitmap;

                    if (softwareBitmap == null && videoMediaFrame != null)
                    {
                        var videoFrame = videoMediaFrame.GetVideoFrame();
                        softwareBitmap = await SoftwareBitmap.CreateCopyFromSurfaceAsync(videoFrame.Direct3DSurface);
                    }

                    if (softwareBitmap != null)
                    {
                        int width = softwareBitmap.PixelWidth;
                        int height = softwareBitmap.PixelHeight;

                        //logger.LogDebug($"Software bitmap pixel fmt {softwareBitmap.BitmapPixelFormat} {width}x{height}.");

                        if (softwareBitmap.BitmapPixelFormat != BitmapPixelFormat.Nv12)
                        {
                            softwareBitmap = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Nv12, BitmapAlphaMode.Ignore);
                        }

                        // Swap the processed frame to _backBuffer and dispose of the unused image.
                        softwareBitmap = Interlocked.Exchange(ref _backBuffer, softwareBitmap);
                        softwareBitmap?.Dispose();

                        using (BitmapBuffer buffer = _backBuffer.LockBuffer(BitmapBufferAccessMode.Read))
                        {
                            using (var reference = buffer.CreateReference())
                            {
                                unsafe
                                {
                                    byte* dataInBytes;
                                    uint capacity;
                                    ((IMemoryBufferByteAccess)reference).GetBuffer(out dataInBytes, out capacity);

                                    lock (_vp8Encoder)
                                    {
                                        byte[] encoderInBuffer = new byte[capacity];
                                        Marshal.Copy((IntPtr)dataInBytes, encoderInBuffer, 0, (int)capacity);

                                        var encodedBuffer = _vp8Encoder.Encode(encoderInBuffer, _forceKeyFrame);
                                        OnLocalVideoEncodedSample.Invoke(_backBuffer);

                                        if (encodedBuffer != null)
                                        {
                                            uint fps = (_fpsDenominator > 0 && _fpsNumerator > 0) ? _fpsNumerator / _fpsDenominator : DEFAULT_FRAMES_PER_SECOND;
                                            uint durationRtpTS = VIDEO_SAMPLING_RATE / fps;

                                            OnVideoSourceEncodedSample.Invoke(durationRtpTS, encodedBuffer);
                                        }

                                        if (_forceKeyFrame)
                                        {
                                            _forceKeyFrame = false;
                                        }
                                    }
                                }
                            }
                        }

                        //_backBuffer?.Dispose();
                    }
                }
            }
        }
    }

 private void VideoEndpoint_OnLocalVideoEncodedSample(SoftwareBitmap softwareBitmap)
    {
        //SoftwareBitmap _backbuffer = null;

        BeginInvoke(new Action(() =>
       {
           if (softwareBitmap.BitmapPixelFormat != Windows.Graphics.Imaging.BitmapPixelFormat.Bgra8 ||
                   softwareBitmap.BitmapAlphaMode != Windows.Graphics.Imaging.BitmapAlphaMode.Premultiplied)
           {
               softwareBitmap = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8,
                       BitmapAlphaMode.Premultiplied);
           }

           int width = softwareBitmap.PixelWidth;
           int height = softwareBitmap.PixelHeight;

          using (BitmapBuffer buffer = softwareBitmap.LockBuffer(BitmapBufferAccessMode.Read))
           {
               using (var reference = buffer.CreateReference())
               {
                   unsafe
                   {
                       byte* dataInBytes;
                       uint capacity;
                       ((IMemoryBufferByteAccess)reference).GetBuffer(out dataInBytes, out capacity);

                       Bitmap bmpImage = new Bitmap((int)width, (int)height,
                               (int)(capacity / height), PixelFormat.Format32bppArgb,
                               (IntPtr)dataInBytes);
                       localVideo.Image = bmpImage;
                   }
               }
           }
       }));
    }
sipsorcery commented 3 years ago

I've wired up the WindowsVideoEndPoint.OnVideoSourceRawSample event now. See the GetStartedVideo demo for example usage.

thornzz commented 3 years ago

Thank you very much. I have implemented this code to Softphone Example now its working with local video. I wont close issue so you can fix bug.

     _sipClients[0].MediaSession.Media.VideoSource.OnVideoSourceRawSample +=
                       (milliseconds, width, height, sample, format) =>

                           BeginInvoke(new Action(() =>
                           {
                               unsafe
                               {
                                   fixed (byte* s = sample)
                                   {
                                       System.Drawing.Bitmap bmpImage = new System.Drawing.Bitmap((int)width,
                                           (int)height, (int)width * 3,
                                           System.Drawing.Imaging.PixelFormat.Format24bppRgb, (IntPtr)s);
                                       localVideo.Image = bmpImage;
                                   }
                               }
                           }));
sipsorcery commented 3 years ago

The bug where the softphone demo was not falling back to a test pattern when no webcam was available has already been fixed, closing.

Shaxzod1311 commented 1 year ago

HI I want to try SipSorcery library to use manage WebRTC into asp net application, clearly i want to build server that handle each peerConnection in may future project. I tried WebRTCGetStarted from examples and i got an error Method 'add_OnVideoSourceRawSampleFaster' in type 'SIPSorceryMedia.Encoders.VideoEncoderEndPoint' from assembly 'SIPSorceryMedia.Encoders, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation My OS Windows 11. I will be very grateful, if you help me for solving this problem