search

twilio / video-quickstart-ios

Twilio Video Quickstart for iOS
https://www.twilio.com/docs/api/video
MIT License
461 stars 178 forks source link

Mitigating Blurriness & Green Screens #378

Closed jvmontes closed 3 years ago

jvmontes commented 5 years ago

Description

Hello there, me again. Thank you for your support earlier on finding solutions we could use with ReplayKit. As we continue to test and provide the best experience we can, as we were doing with our earlier implementation, we now see a defect where participant screens intermittently flash bright green. How can we avoid the green screen issue? And how can we get blurriness to clear quickly and consistently?

Steps to Reproduce

  1. Start the video stream
  2. Navigate through different parts of the application (using segues, general updates from UIControls)
  3. Observe the participant

Code

We are leveraging the existing ReplayKitVideoSource.swift provided to stream our meeting.

Expected Behavior

Participants should never see a flash of green on their screen. Blurry participant screens should also be consistently resolved quickly.

Actual Behavior

Participants intermittently see green flashing on their screen. Participant screens may stay blurry for up to several seconds, or consistently appear blurry. May go back and forth from being blurry, to being clear, to being blurry again for several seconds, and never quite reach the level of pristine display it may initially have, or may take several seconds to display the level of pristine picture quality.

Reproduces How Often

10% Green Screen. Have noticed that it hasn't happened as much when I lowered kDesiredFrameRate to 30 from 120.

30% Blurriness Issue.

Versions

All relevant version information for the issue.

Video iOS SDK

2.7.0

Xcode

10.1

iOS Version

11.3.1

iOS Device

A1584 iPad Pro

BlurryUI

ClearUI

ceaglest commented 5 years ago

Hi @jvmontes,

Sorry that we did not get a chance to cover your question today, but we will dig into it tomorrow.

Off the top of my head, there are several reasons why the resolution of the video might drop, including constrained sender or receiver bandwidth. If its very important that spatial resolution be maintained then you can use the screencast mode in ReplayKitVideoSource.

10% Green Screen. 30% Blurriness Issue.

Which video codec are you using, and in what kind of Room?

I will see if @paynerc can help on this one, as he is pretty familiar with your use case.

Best, Chris

jvmontes commented 5 years ago

Hi @ceaglest ,

Thank you for your reply. I feel silly now, it seems that setting screencastUsage fixed our blurriness issue, thanks so much.

We still intermittently see a green screen mask the entire display before clearing up shortly after. In my opinion, this isn't a showstopper but definitely a jarring experience on the participant's end. Have you seen anything regarding this or have ways to mitigate it?

Thanks again, we're almost at the finish line!! 🙌

paynerc commented 5 years ago

Hey @jvmontes,

As @ceaglest mentioned, it would be helpful to know a couple things about your setup. Specifically,

Also, and room sids for when you have experienced this issue could also be useful. You could send those to the support email, reference me and this GitHub issue and it should find its way to me.

Thanks!

Ryan

jvmontes commented 5 years ago

Hi Ryan,

We are using the VP8 Codec. I'm not sure if we are using Group or Peer to Peer rooms, I've looked around in our front-end and back-end code and can't find anything. Do you know how I can find this information? I am able to reproduce the green screen issue by just connecting a single participant.

I've sent an email to Alex from the support team with the room SID.

Thanks! Jorge

paynerc commented 5 years ago

@jvmontes,

Just a follow up on this ticket here, I have responded to you through the support system. We can leave this issue open in the meantime until we determine the issue and then post a public reply about what we find. We can continue to work through your open support ticket until that time.

Thanks!

Ryan

paynerc commented 4 years ago

@jvmontes,

I know that we had worked offline through the support ticket. Are you still seeing issues with this or is it OK to close this issue?

Thank you,

Ryan

GPITwilioMonitor commented 4 years ago

Hi all,

I'm part of a new team that has taken ownership of this feature, apologies for the silence these past couple months. Can we re-open this ticket? I can confirm that the green screen issue is still present for our application.

To keep you up-to-date on some of the specs:

iOS codebase (this is where we broadcast from) and devices:

We use a web browser for the participants (which receives the video stream):

The issue:

GPITwilioMonitor commented 4 years ago

I've also opened a ticket via the Twilio support center regarding this issue (#4112106).

Here is a screenshot of the green screen that is shared within the ticket: https://p14.zdusercontent.com/attachment/5194/dpeBEmwm2UzCjrAeOtYall80Q?token=eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..eN4-NJi3j_loLEzqcYLjoQ.Xo2YOj9AzpHy9Vl9l9YrX52vRyJPLtl4QUT6X-EHwZvoiLPFNfSedMkWjpUet2hMCzxIdGp4RBd4walD90wKMiINh161O1BwG_flkP43-qdn8MOTXnOG3jM0fA-bjdic8WSCKq8b3rCw1CLWByli_yJUZk1C51fQxo_NCNAmmrg3ppyUnKhhXFLB-9-5I-wgOX_V4qisynLfykluWMYqBb6Qx_T-pZ7jB5WVBiEPrRCEBZ1l8HUTTWix3jKw0gnuLKm-uT4QCHiWqVsHh3I7Rg.RmxyehqI1ygKG8iDp9bG2A

paynerc commented 4 years ago

@GPITwilioMonitor,

I'm sorry you are still experiencing this issue. I will reopen this ticket and review the support ticket you mentioned above. I will get back to you shortly.

Ryan

piyushtank commented 4 years ago

@GPITwilioMonitor - Thanks for reaching out. Yes, we are able to reproduce the bug using our ReplayKit Sample app. We think the problem happens due to the packet loss. Our observation is problem occurs when frame rates supplied by the VideoSource spikes. The problem may disappear by tweaking the frame rate to a higher number say 20fps or a higher number, the downside is it may cause higher cpu utilization and higher battery usage - you may want to try it out to see if you see any improvements.

On our side, we are investigating if our infrastructure media engine or any improvements needs to be made in our ReplayKitVideoSource. I will keep you posted.

Let me know if you have any questions.

GPITwilioMonitor commented 4 years ago

Hi @piyushtank - I'm glad you were able to reproduce the issue on your end. In the past we did experiment with differing frame rates - nothing effectively eliminated the green screen flash.

I also should call out: we don't actually use any of the retransmission logic. We encountered a crash a while back from within the dispatchRetransmissions function. Additionally, the ExampleCoreAudioDeviceGetCurrentQueue which populates the videoQueue variable is using a long-deprecated API to retrieve the current queue. For these two reasons we decided to implement our own mechanism to ensure continuous frames are being sent, and we cut out all of the code from ReplayKitVideoSource relating to retransmissions (including the kFrameRetransmitIntervalMs variable you mentioned above).

I can share our version of ReplayKitExampleCode.swift, what's your email address so that I can send this to you privately?

Thanks, Erin

piyushtank commented 4 years ago

@GPITwilioMonitor Sure, I would be happy to take a look at your version of ReplayKitExampleCode.swift. My email address is ptank@twilio.com.

piyushtank commented 4 years ago

@GPITwilioMonitor We did some investigation on our side - the green screen issue occurs because of the packet loss. This could happen when send bitrate spikes. If you set the minimum fps, it may help little but not much as the encoder optimizes bitrate usage when the still screen image is being sent out upon a higher fps. We think setting the minimum bitrate parameter should help. I have created feature request to add APIs to set screencast min bitrate in our SDK.

For now, increasing the minimum fps is the best options which unfortunately will not solve the problem completely but may help little.

paynerc commented 3 years ago

@GPITwilioMonitor,

Are you still experiencing issues related to this ticket?

Ryan

GPITwilioMonitor commented 3 years ago

Hi Ryan, thanks for checking in. We are actively using the Group Room feature in our production release and have not heard complaints about the green screen.

Mike

paynerc commented 3 years ago

Thanks wonderful news. Thanks for responding. I will close out this issue.

Ryan