WebKit Bug 180748 - Audio sometimes fail to capture in WebRTC

[theDude30]

• [ X] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [ X] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [ ] I verified that the Quickstart application works in my environment.

Code to reproduce the issue:

let constraints={video:true,audio:true}; Video.createLocalTracks(constraints).then((localTracks)=> { var localMediaContainer = document.getElementById('localVideoContainer'); localTracks.forEach((track) =>{ let myElement=track; localMediaContainer.appendChild(myElement.attach()); this.avRoom.localParticipant.publishTrack(track); }); });

Expected behavior: Sound should be heard.

Actual behavior: On all PC,android phones, the sound is heard. on safari 11(iphone,ipad, and on osx) no sound output is heard There are no console exception.

See Logs below: SAFARI:

[Warning] 2018-01-25 (6) (main.js, line 57591) "15:04:43.333Z" "|" "WARN" "in" "[connect #1]:" "This release of twilio-video.js includes experimental support for Safari 11 and newer. Support for Safari is \"experimental\" because, at the time of writing, Safari does not support VP8. This means you may experience codec issues in Group Rooms. You may also experience codec issues in Peer-to-Peer (P2P) Rooms containing Android- or iOS-based Participants who do not support H.264. However, P2P Rooms with browser-based Participants should work. Please test this release and report any issues to https://github.com/twilio/twilio-video.js" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.355Z" "|" "INFO" "in" "[connect #1]:" "Connecting to a Room" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.356Z" "|" "INFO" "in" "[NTSIceServerSource #1]:" "Created a new NTSIceServerSource" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.356Z" "|" "INFO" "in" "[connect #1]:" "LocalTracks were not provided, so they will be acquired automatically before connecting to the Room. LocalTracks will be released if connecting to the Room fails or if the Room is disconnected" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.357Z" "|" "INFO" "in" "[createLocalTracks #1]:" "Neither audio nor video requested, so returning empty LocalTracks" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.359Z" "|" "INFO" "in" "[LocalParticipant #1]:" "Created a new Participant" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.359Z" "|" "INFO" "in" "[connect #1]:" "Getting ICE servers" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:43.359Z" "|" "INFO" "in" "[NTSIceServerSource #1]:" "Starting" [Log] Loaded p (client, line 1694) [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:44.347Z" "|" "INFO" "in" "[NTSIceServerSource #1]:" "Got ICE servers: [{\"username\":\"83d965157dee0f8e8617959da33c317eadd9bd58ead7c50494ffa3ebea3fdcd8\",\"credential\":\"oEqSYlsnzL9ERN3OC8O+B5d5jyBlh/vTBZzhrYFOSho=\",\"urls\":\"turn:global.turn.twilio.com:3478?transport=udp\"},{\"username\":\"83d965157dee0f8e8617959da33c317eadd9bd58ead7c50494ffa3ebea3fdcd8\",\"credential\":\"oEqSYlsnzL9ERN3OC8O+B5d5jyBlh/vTBZzhrYFOSho=\",\"urls\":\"turn:global.turn.twilio.com:443?transport=tcp\"}]" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:44.348Z" "|" "INFO" "in" "[NTSIceServerSource #1]:" "Getting ICE servers again in 3600 seconds" [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:44.348Z" "|" "INFO" "in" "[connect #1]:" "Got ICE servers" [Warning] Warning: This browser does not support audio output selection. (twilio.min.js, line 35) [Log] [Device] Setting up PStream (twilio.min.js, line 105) [Log] [WSTransport] Opening socket (twilio.min.js, line 105) [Log] [WSTransport] attempting to connect (twilio.min.js, line 105) [Warning] Device.sounds is deprecated and will be removed in the next breaking release. Please use the new functionality available on Device.audio. (twilio.min.js, line 105) [Info] 2018-01-25 (7) (main.js, line 57591) "15:04:45.302Z" "|" "INFO" "in" "[Room #1: RMa49677de27320c0639c1f4b96f2c3605]:" "Created a new Room:" "c02880db-dc7e-4726-8e63-e1b0f46246b4" [Info] 2018-01-25 (7) (main.js, line 57591) "15:04:45.302Z" "|" "INFO" "in" "[connect #1]:" "Connected to Room:" "[Room #1: RMa49677de27320c0639c1f4b96f2c3605]" [Info] 2018-01-25 (7) (main.js, line 57591) "15:04:45.303Z" "|" "INFO" "in" "[connect #1]:" "Room name:" "c02880db-dc7e-4726-8e63-e1b0f46246b4" [Log] [WSTransport] Socket opened (twilio.min.js, line 105) [Log] [PStream] Setting token and publishing listen (twilio.min.js, line 105) [Log] [Device] Stream is ready (twilio.min.js, line 105) [Log] Twilio.Device.ready (pcToPhone.html, line 60) [Info] 2018-01-25 (6) (main.js, line 57591) "15:04:54.630Z" "|" "INFO" "in" "[RemoteParticipant #2: PA0c32473629a80af692ba6bb405dd039c]:" "Created a new Participant: NastyXanderVail" [Info] 2018-01-25 (7) (main.js, line 57591) "15:04:54.631Z" "|" "INFO" "in" "[Room #1: RMa49677de27320c0639c1f4b96f2c3605]:" "A new RemoteParticipant connected:" RemoteParticipant {_events: Object, _maxListeners: undefined, _instanceId: 2, _log: Log, _signaling: RemoteParticipantV2, …} [Log] Loaded p (client, line 1694) [Info] 2018-01-25 (7) (main.js, line 57591) "15:04:59.681Z" "|" "INFO" "in" "[RemoteParticipant #2: PA0c32473629a80af692ba6bb405dd039c]:" "Added a new AudioTrack:" "2755ce63-5930-4fed-ad4b-e88749275491" [Info] 2018-01-25 (7) (main.js, line 57591) "15:04:59.689Z" "|" "INFO" "in" "[RemoteParticipant #2: PA0c32473629a80af692ba6bb405dd039c]:" "Added a new VideoTrack:" "60fc3540-0753-4d62-b0c5-ea16fb637b35"

Chrome:

2018-01-25 15:04:52.795Z | INFO in [connect #1]: Connecting to a Room main.js:57591 2018-01-25 15:04:52.797Z | INFO in [NTSIceServerSource #1]: Created a new NTSIceServerSource main.js:57591 2018-01-25 15:04:52.798Z | INFO in [connect #1]: LocalTracks were not provided, so they will be acquired automatically before connecting to the Room. LocalTracks will be released if connecting to the Room fails or if the Room is disconnected main.js:57591 2018-01-25 15:04:52.799Z | INFO in [createLocalTracks #1]: Neither audio nor video requested, so returning empty LocalTracks main.js:57591 2018-01-25 15:04:52.802Z | INFO in [LocalParticipant #1]: Created a new Participant main.js:57591 2018-01-25 15:04:52.802Z | INFO in [connect #1]: Getting ICE servers main.js:57591 2018-01-25 15:04:52.803Z | INFO in [NTSIceServerSource #1]: Starting main.js:57591 2018-01-25 15:04:54.650Z | INFO in [RemoteParticipant #2: PAfa8e065fcb638230a082b9e4c06a3f17]: Created a new Participant: JoltinTammyJamestown main.js:57591 2018-01-25 15:04:54.650Z | INFO in [Room #1: RMa49677de27320c0639c1f4b96f2c3605]: A new RemoteParticipant connected: RemoteParticipant main.js:57591 2018-01-25 15:04:54.652Z | INFO in [Room #1: RMa49677de27320c0639c1f4b96f2c3605]: Created a new Room: c02880db-dc7e-4726-8e63-e1b0f46246b4 main.js:57591 2018-01-25 15:04:54.653Z | INFO in [connect #1]: Connected to Room: [Room #1: RMa49677de27320c0639c1f4b96f2c3605] main.js:57591 2018-01-25 15:04:54.653Z | INFO in [connect #1]: Room name: c02880db-dc7e-4726-8e63-e1b0f46246b4 main.js:57591 2018-01-25 15:04:59.447Z | INFO in [LocalParticipant #1: PA0c32473629a80af692ba6bb405dd039c]: Added a new AudioTrack: 2755ce63-5930-4fed-ad4b-e88749275491 main.js:57591 2018-01-25 15:04:59.448Z | INFO in [LocalParticipant #1: PA0c32473629a80af692ba6bb405dd039c]: Added a new LocalAudioTrack: 2755ce63-5930-4fed-ad4b-e88749275491 main.js:57591 2018-01-25 15:04:59.449Z | INFO in [LocalParticipant #1: PA0c32473629a80af692ba6bb405dd039c]: Added a new VideoTrack: 60fc3540-0753-4d62-b0c5-ea16fb637b35 main.js:57591 2018-01-25 15:04:59.450Z | INFO in [LocalParticipant #1: PA0c32473629a80af692ba6bb405dd039c]: Added a new LocalVideoTrack: 60fc3540-0753-4d62-b0c5-ea16fb637b35 main.js:57591 2018-01-25 15:04:59.881Z | INFO in [LocalParticipant #1: PA0c32473629a80af692ba6bb405dd039c]: Created a new VideoTrackPublication: MT3f0b9331e24b911b07d9a9e873bc4a82 main.js:57591 2018-01-25 15:04:59.882Z | INFO in [LocalParticipant #1: PA0c32473629a80af692ba6bb405dd039c]: Created a new AudioTrackPublication: MT89c53fc6611db01f0026a1fe6a4df592

TODO

Software versions:

• [ X] Browser(s):Chrome 63.0.3239.132 (Official Build) (64-bit) , Safari Version 11.0.3 (13604.5.6)
• [X ] Operating System:OSX 10.13.3
• [ X] twilio-video.js: 1.7.0
• [ X] Third-party libraries (e.g., Angular, React, etc.): ReactJS

[markandrus]

@theDude30 thank you for the report. I will attempt to reproduce. Safari 11.0.3 was released January 23, 2018. May laptop is still using 11.0.2. I will update and test it out.

[markandrus]

I can reproduce with Safari 11.0.3. Safari 11.0.2 worked fine for me.

[markandrus]

Actually, I think the issue is intermittent. When the issue manifests, I see packetsReceived, bytesReceived, and audioLevel are zero for remote audio tracks. I saw this in 11.0.3. I restarted Safari and tried some different tests and it seemed to resolve. Very strange... I'm still investigating.

[markandrus]

So, after I restarted Safari, the issue went away and I am unable to reproduce in 11.0.3. @theDude30 can you see if restarting fixes the behavior? I'm still not sure how to trigger reliably.

[theDude30]

i've restarted my MAC computer and it now works on MAC (safari). However, the problem still occur on my iPad even after restarting. the iPad version is: iOS 11.2.1 Safari 11

[theDude30]

UPDATE: if the iOS device starts to publish audio/video (2 way audio/video), the sound works. If the iOS only receive audio/video from the PC without starting it's own audio/video, the iOS not getting sound.

[theDude30]

on some occasions 2 way audio video also does solve the problem. any update on this? this is critical bug. we have clients constantly complaining about this.

[markandrus]

@theDude30,

If the iOS only receive audio/video from the PC without starting it's own audio/video, the iOS not getting sound.

Thanks for the additional information. I'll try to repro with this.

[markandrus]

Unfortunately, this is a WebKit bug. It can be reproduced without twilio-video.js, without WebRTC, using just getUserMedia sample code. I haven't seen a workaround yet, but it's being actively discussed on these two tickets:

• Bug 179964 - Camera/microphone acquisition false positive after receiving phone call
• Bug 180748 - Audio sometimes fail to capture in WebRTC

[theDude30]

It looks like the bug still exist in the new version of ios. However, in the following thread there are few workarounds in order to resolve this temporary. Can you please have a look and let me if it's possible to implement this on the twilio library: https://bugs.webkit.org/show_bug.cgi?id=180748

[markandrus]

Thanks for the headsup @theDude30. Here is the gist of the workaround (note that the location.reload() is not a good solution since you may not want to reload the page):

You can do a getUserMedia call. If you do not get any audio, you can do the following:

1. stop the audio track
2. call getUserMedia to get audio
3. Redo step 1 and 2 and you should hopefully get some audio Let me know if that works for you.

This is unfortunate. Chrome had a bug like this for a long time, and Twilio worked around it in this way in our Voice SDK. It requires monitoring the audio levels on the track, but the issue is that you have to delay returning a track since

• Looping through Steps 1–3 takes time,
• You also have to sample the audio track for a couple hundred milliseconds to get reliable results, and

• Even then an intentionally-muted audio track may cause you to loop forever...

  We could potentially workaround this in our createLocalTracks implementation, although it's unfortunate. We should definitely add this to COMMON_ISSUES.md.

[quentinhayot]

Is it something you guys are working on? Can we expect the workaround to be implemented soon? (No pressure, it's just so we know what to expect :) )

[markandrus]

@quentinhayot We could implement this workaround behind a flag. Basically, anywhere an audio track can be requested, we allow passing a new option, workaroundWebKitBug180748, defaulting to false. If true, execute the workaround. For example,

createLocalAudioTrack({
  workaroundWebKitBug180748: true
});

createLocalTracks({
  audio: {
    workaroundWebKitBug180748: true
  }
});

connect(token, {
  audio: {
    workaroundWebKitBug180748: true
  }
});

It is ugly, but this is basically what we did for Twilio Client (twilio.js) back when WebRTC Issue 3940 was a thing. We added an option, chrome3940Workaround. So there's precedent... WDYT? We could implement something like this.

[quentinhayot]

That would be awesome. Period.

[markandrus]

@quentinhayot OK, we have scheduled the work 👍

[theDude30]

any update on this? i'm still having issues sound issues on iOS

[theDude30]

I've tried using the "workaroundWebKitBug180748" option and it worked but the biggest problem that it broadcasts the microphone which i really don't want in that point. Any thoughts on how to resolve this?

[markandrus]

Hi @theDude30,

any update on this? i'm still having issues sound issues on iOS

Yes, I've started implementing the workaround. I'll share a PR this week.

I've tried using the "workaroundWebKitBug180748" option and it worked but the biggest problem that it broadcasts the microphone which i really don't want in that point.

Sorry, how do you mean? Is this because you attach to the AudioContext destination, which defaults to playing out? I'm still testing myself. EDIT: Another option that may avoid play-out is to use two RTCPeerConnections in loopback and monitor getStats.

[markandrus]

@quentinhayot @theDude30 I've implemented the workaroundWebKitBug180748 proposal here: https://github.com/twilio/twilio-video.js/pull/329 Do either of y'all mind trying it out? You'd have to build the branch until we cut a proper RC. I'm having a bit of troubling validating the implementation, because I can't always trigger the bug.

[theDude30]

I've built the branch and did tests. User 1: OSX Chrome 67 User 2: iPad air, ios 11.4, safari.

Scenario 1: Users connect using the workaround

Video.connect(data.token, {name: room,audio: { workaroundWebKitBug180748: true },video:false }).then(roomJoined, function (error) {

• 1 user broadcast video using: Twilio.Video.createLocalTracks({video: true,audio: true})

Result: Ipad user is prompted to broadcast his microphone, once allowed the ipad could see and hear the chrome user and everything is working properly.

Scenario 2: Users connect normally and broadcast with the workaround Video.connect(data.token, {name: room,audio: false,video:false }).then(roomJoined, function (error) {

Twilio.Video.createLocalTracks({video: true,audio: { workaroundWebKitBug180748: true }}) Result: No microphone allow prompt in the connect and the iPad user could not hear the chrome user(he could see the video but could not hear anything).

Conclusions:

1. In order to make it work users must connect with the workaroundWebKitBug180748: true .
2. It make no effect when using workaroundWebKitBug180748: true in createLocalTracks.
3. Although the workaround parameter works, it prompts the iPad user to allow the microphone which is not really broadcasting the microphone. I know that this is how the workaround works and it's because of apple but i'm wondering if there is anything else that could have implemented to not showing this prompt to the user.

[markandrus]

@theDude30 it's strange it doesn't work for you in the second case. It should essentially be the same as the first. Note, however, it won't always workaround the issue. Nevertheless, we'll release it in 1.11.0.

3. Although the workaround parameter works, it prompts the iPad user to allow the microphone which is not really broadcasting the microphone. I know that this is how the workaround works and it's because of apple but i'm wondering if there is anything else that could have implemented to not showing this prompt to the user.

What do you mean by "prompts the iPad user to allow the microphone"? The workaround is specifically for the case where you are attempting to publish audio, so it must prompt for the microphone anyway.

[raduanastase]

For me, when I'm using workaroundWebKitBug180748 flag, the video stops working, but the audio works fine. Any news on when will this be fixed?

[manjeshbhargav]

Hi @raduanastase ,

Thanks for writing in with your issue. When you say the video stops working, do you mean that the local video preview also freezes along with the video on the remote size? Can you share a Room SID and a screenshot of the behavior if possible?

Thanks,

Manjesh Malavalli JSDK Team

[raduanastase]

After fixing the errors in my code, it started working :) Sorry about that. The video didn't freeze, it didn't start at all.

[EugeneNF]

https://bugs.webkit.org/show_bug.cgi?id=180748 got fixed Sept 2019

[manjeshbhargav]

Hi all,

I'm closing this issue since the webkit bug has been fixed.

Thanks,

Manjesh Malavalli JSDK Team