regarding Audio video sync

[suganthikarthick]

Hi,

We are getting the audio and video working in our platform with openwebrtc, but still the video is 3 sec delayed when compared to the audio. Is there any solution for making the audio and video to sync? It would be a great help if your give your suggestion on this.

Thanks

[sdroege]

Most likely fixed by https://github.com/EricssonResearch/openwebrtc/issues/240

[suganthikarthick]

In our case, we are using the custom gst audio (autoaudiosrc) for audio and local src video (v4l2src) and the media is transferring as VP8 encoded.

In #240 i am seeing that we need to maintain the similar timestamp in the pipeline. Can you explain us how to apply that?

Thanks,

[suganthikarthick]

Hi, We modified certain parameters inside the owr_payload.c - for vp8 encoder. But still the delay is there. Since the video is delayed, there is no sync between audio and video while sending it from the openwebrtc application to the other end. In x86, it is fine. But when we try to use it in our ARM target imx6, we are facing this delay issue. Does anyone has suggestion about reducing the video delay or making the audio/video to sync in ARM platform?

[superdump]

240 has now been merged. It improved latency significantly on desktop (Mac OS X and Linux) but has not improved it on iOS. I have not tested on other platforms yet. It's a top-priority issue to address. Please bear with us while we investigate and fix this.

[suganthikarthick]

Hi, In the libvpx library, we have enabled the CONFIG_REALTIME_ONLY and CONFIG_ONTHEFLY_BITPACKING. After that we tried to use the gst-launch command for sending the video to another machine, there is no delay now. But still, a delay of 4s is there when running our application. We played around with the vp8 encoding also. What else may create this delay of 4s. It would be of great help if any suggestions are there.

[superdump]

I suspect it is a more complicated issue in GStreamer or in the GStreamer elements. It is being investigated.

[suganthikarthick]

Ok, but it is working perfectly with the following gstreamer pipeline...

gst-launch-1.0 v4l2src ! videoconvert ! video/x-raw,width=640,height=360 ! vp8enc cpu-used=16 ! rtpvp8pay ! udpsink host=192.168.101.186 port=5000?

Thanks.

[superdump]

The pipelines used in OpenWebRTC are more complex.

ICE and DTLS take time to complete before data can actually be sent. We suspect that some video processing is starting before those complete and then when the data actually gets sent, it is sent in real time instead based on the buffer timestamps of as quickly as possible. This could cause the data to get buffered up in the pipeline and never catch up. This is then observed as a delay on the receiving side.

One possible thing to try to perhaps confirm this would be to change the following to if(true): https://github.com/EricssonResearch/openwebrtc/blob/master/transport/owr_transport_agent.c#L1067

[suganthikarthick]

Hi Robert, Thanks for the response. We tried that. But still the delay is there. But this delay is not happening with X86.. In that it is fine.

[suganthikarthick]

Hi, We called the function "owr_media_session_set_send_source", only after sending the answer.

That is, Once the DTLS and ICE are completed, we are sending the answer. After sending the answer, we are calling the "owr_media_session_set_send_source". But still we are having the same delay.

[godivyam]

any updates ?

[superdump]

Closing this issue as the delay issue is being tracked in #451. A fix for H.264 has been made but there are still some issues with VP8 and libnice.

@suganthikarthick : are your patches / modifications and instructions for using OpenWebRTC on Freescale i.MX6 available anywhere? They are certainly interesting for others using the same platform.

[suganthikarthick]

Currently we are working for increasing the gst-launch performance when using vp8enc on IMX6. We are currently using the resolution 1920x1080. Upto 1280x720, the video is coming without any delay. But for the full HD resolution, we are getting poor and distorted video. Does anybody has any idea about how to optimize the vp8enc better for full HD resolution?

[suganthikarthick]

@suganthikarthick : are your patches / modifications and instructions for using OpenWebRTC on Freescale i.MX6 available anywhere? They are certainly interesting for others using the same platform.

Hi, we are using the same versions of all the libraries that are used in the cerbero build. We have taken all the packages and build using the Yocto build for IMX6 platform.

We haven't faced any issues on using the test client application on the IMX6 platform. Only thing we are struggling now is for making the performance improvement for VP8 video encoding on this platform.

Also we are planning to use imxvpuenc_h264 hardware encoder in the openwebrtc. We need your help on using this element for encoding.

We are using the custom gst APIs for the getting the different video sources and audio sources also.

Thanks.

[superdump]

Poor and distorted video... that's a different issue to delayed video. Can you be more specific? Does it look like the hardware cannot keep up with encoding 1080p VP8 in realtime? Does it look like packet loss which produces corruption in the resulting video on the receiver side? Is the video delayed?

[suganthikarthick]

Here, our video source is of H264 encoded format. So, we are decoding that and making it as raw video and with again encoding with vp8 for sending it for openwebrtc call.

Seems there is a packet loss. We are not sure whether it is because of the hardware or due to vp8enc software encoder on this platform.

[superdump]

Can you open a separate issue about that please? So we can keep this one focused on the delay issue.

[suganthikarthick]

Ok.