search

twilio / twilio-video.js

Twilio’s Programmable Video JavaScript SDK
https://www.twilio.com/docs/video/javascript
Other
571 stars 217 forks source link

Pixelated Video Stream from iOS/Safari #1055

Closed amadoru closed 4 years ago

amadoru commented 4 years ago

When connecting to the rooms from iOS devices, we are most frequently noticing that the video stream that the outbound video stream is pixelated to the receiving party. While this is mostly seen on outbound video from iOS/Safari, we also occasionally notice this on other devices/browsers as well.

When connecting to rooms we have tested this on Small Group rooms set to gll media region and tested on few regions. We have tested with both VP8 and H.264 video codecs individually.

Issue is reproducible in both the Twilio video react app as well as on our custom implementation.

Expected behavior:

Video is expected to be of higher quality and is non-pixelated.

Software versions:

Note: This issue may be related to https://github.com/twilio/twilio-video.js/issues/983

manjeshbhargav commented 4 years ago

Hi @amadoru ,

Thanks for writing in with your issue. Poor video quality can be due to many things, one of them being packet loss due to poor network conditions. To start off, please provide a couple of Room SIDs where you observed this behavior. Also, it would be great if you can share the ConnectOptions object you are passing to connect().

Thanks,

Manjesh Malavalli JSDK Team

amadoru commented 4 years ago

Hi @manjeshbhargav,

Thank you for the response.

Yes, I understand the perceived quality can be due to many parameters around the network conditions. My objective is to improve the perceived quality of our Twilio Video backed system to have the perceived quality similar to what other major WebRTC based implementations offer (eg: Goog. Meet).

SID: RM89f6e0fa8174cff3d0ffb94c8a283739 Codec: VP8 Connect Options: { audio: true, video: { width: 1280, height: 720, frameRate: 18 }, maxAudioBitrate: 16000, networkQuality: { local:1, remote: 1 } }

Please do let me know if I can assist further with more debug/other information.

Thank you.

manjeshbhargav commented 4 years ago

Hi @amadoru ,

I took a look at our logs for the Room SID you shared, and there seems to have been significant packet loss for the Chrome Participant's video while it is being transmitted to the iOS Participant, most likely due to network issues.

I recommend that you take advantage of a feature in Group Rooms called Bandwidth Profiles, which will properly allocate bandwidth to video tracks based on your network's capacity so that packet loss will be minimized, and therefore pixelation will be reduced. Please use the following ConnectOptions settings: ConnectOptions for desktop browsers and ConnectOptions for mobile browsers.

Please let me know if this helps.

Thanks,

Manjesh Malavalli JSDK Team

Balasnest commented 4 years ago

1059 Facing same issue like this.

amadoru commented 4 years ago

Hi @manjeshbhargav,

Thank you for the recommendation.

I have previously tried using the bandwidth profiles and I have again tried with the bandwidth profiles. In all cases, I have not seen any noticeable difference. Our use case is 1:1 video calls, and below is one of the connectionOptions that I have tried based on your recommendation.

  const connectionOptions: ConnectOptions = {
    audio: true,
    video: isIOS? { height: 480, frameRate: 24, width: 640 } : { height: 720, frameRate: 24, width: 1280 },
    maxAudioBitrate: 16000,
    networkQuality: { local: 1, remote: 1 },
    bandwidthProfile: {
      video: {
        maxSubscriptionBitrate: 2500000,
        renderDimensions: {
          high: {height:1080, width:1920},
          standard: {height:720, width:1280},
          low: {height:176, width:144}
        }
      }
    }

Furthermore, the pixelation primarily happens on the video captured on iOS and displayed on desktop browser (in this case, Chrome). On the other direction (Capture video on Chrome/Mac and view on iOS), video quality is lower but a pixelation is almost unnoticeable.

Many of the tests we have run were on different networks, and these networks generally provide decent connection. During the tests I have performed today, based off chrome WebRTC interals, I am seeing ~1-2 packets lost per minute on average, yet see a noticeable pixelation. (Furthermore, this issue is reported by several clients who have highly stable connections and are relatively closer to media region than I am).

Please let me know your recommendation.

Thank you.

amadoru commented 4 years ago

Hi @manjeshbhargav, do you have any update to this? Thanks.

manjeshbhargav commented 4 years ago

Hi @amadoru ,

Please add the following to your ConnectOptions:

options.bandwidthProfile.video.mode = 'collaboration';
options.preferredVideoCodecs = [{ codec: 'VP8', simulcast: true }];

Our tests and experiences from other customers have shown that VP8 simulcast has better overall video quality. Please try it out and let me know what you find.

Thanks,

Manjesh Malavalli JSDK Team

amadoru commented 4 years ago

Hi @manjeshbhargav,

Thank you for the suggestion. We have tested this suggestion and based on our testing, the change of video mode to Collaboration didn't make any visible change for us. Further, the use of code with Simulcast has significantly degraded the video quality, contrary to the expectation of increased quality.

Thanks you.

manjeshbhargav commented 4 years ago

Hi @amadoru ,

Please send me some Room SIDs which use the new configuration. Also, it would be great if you can send a screenshot of how the mobile client's video looks like from the other client's point of view.

Thanks,

Manjesh Malavalli JSDK Team

manjeshbhargav commented 4 years ago

@amadoru ,

I'm closing this issue since I have not heard from you since a long time. Please feel free to re-open if you still need any help in this matter.

Thanks,

Manjesh Malavalli JSDK Team