samgabriel commented 3 years ago

Please read first!

Please use Public Google Group (mailing list) for general technical discussions and questions.

[X ] I have used Google with the error message or bug in association with the library and Cordova words to make sure the issue I'm reporting is only related to iOSRTC.
[X ] I have provided steps to reproduce (e.g. sample code or updated extra/renderer-and-libwebrtc-tests.js file).
[ X] I have provided third party library name and version, ios, Xcode and plugin version and adapter.js version if used.

Note: If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue.

Versions affected

Cordova version (e.g 8.0):
Cordova iOS version (e.g 4.6.4):
Plugin version (e.g 6.0.15):
iOS version (e.g 13.3):
Xcode version (e.g 11.3.1):
WebRTC-adapter version (e.g. 7.4.0):
WebRTC Framework version (Twilio 2.7.2):

Description

After upgrading the tester to use the latest 6.0.15 now no videos are showing from the remote user at all and local videos are not showing on the remote.


<!DOCTYPE HTML>
<html>
<head>
    <title>
        Twilio Video Room
    </title>
    <meta name="viewport" content="viewport-fit=cover, width=device-width, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no">
    <meta http-equiv="Content-Security-Policy"
          content="default-src 'self' data: gap: wss://* https://* 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; media-src *">
    <script type="text/javascript" src="cordova.js"></script>
</head>
<body>
<br><br><br>
<div id="local-media" style="border: red 1px solid;"></div>
<div id="remote-media"></div>
<button onclick="toggleVideo()">Toggle video</button>
<button onclick="leaveMeeting()">Leave meeting</button>
<button onclick="startMeeting()">Start meeting</button>
<button onclick="switchCamera()">Switch camera</button>
<script type="text/javascript">

const token = '';
const roomName = 'test';
const scriptUrls = [];
let videoTrack = null;
let audioTrack = null;
let room = null;
let facing = true;

function loadScript(scriptUrl) {

  if (scriptUrls[scriptUrl]) {
    return Promise.resolve(scriptUrl);
  }

  return new Promise(function (resolve, reject) {
    // load adapter.js
    var script = document.createElement("script");
    script.type = "text/javascript";
    script.src = scriptUrl;
    script.async = false;
    document.getElementsByTagName("head")[0].appendChild(script);
    script.onload = function () {
      scriptUrls[scriptUrl] = true;
      console.debug('loadScript.loaded', script.src);
      resolve(scriptUrl);
    };
  });
}

async function startMeeting() {

  videoTrack = await getVideoTrack();
  audioTrack = await Twilio.Video.createLocalAudioTrack();
  const localMediaContainer = document.getElementById('local-media');
  localMediaContainer.appendChild(videoTrack.attach());
  localMediaContainer.appendChild(audioTrack.attach());
  console.log(videoTrack, audioTrack);

  Twilio.Video.connect(token, {
    tracks: [videoTrack, audioTrack],
    name: roomName,
    sdpSemantics: 'plan-b',
    bundlePolicy: 'max-compat'
  }).then(_room => {
    room = _room;
    console.log(`Successfully joined a Room: ${room}`);

    // Attach the Tracks of the Room's Participants.
    room.participants.forEach(function (participant) {
      console.log("Already in Room: '" + participant.identity + "'");
      participantConnected(participant);
    });

    room.on('participantConnected', participant => {
      console.log(`A remote Participant connected: ${participant}`);
      participantConnected(participant);
    });

    room.on('participantDisconnected', participant => {
      console.log(`A remote Participant connected: ${participant}`);
      participantDisconnected(participant);
    });

  }, error => {
    console.error(`Unable to connect to Room: ${error.message}`);
  });

  function participantConnected(participant) {
    console.log('Participant "%s" connected', participant.identity);
    const div = document.createElement('div');
    div.id = participant.sid;
    participant.tracks.forEach((publication) => {
      console.log('subbing to existing publication', publication);
      trackSubscribed(div, publication);
    });

    participant.on('trackPublished', (publication) => {
      trackSubscribed(div, publication)
    });
    participant.on('trackUnpublished', trackUnsubscribed);

    document.getElementById('remote-media').appendChild(div);
  }

  function participantDisconnected(participant) {
    console.log('Participant "%s" disconnected', participant.identity);

    var div = document.getElementById(participant.sid);
    if (div) {
      div.remove();
    }
  }

  function trackSubscribed(div, publication) {
    console.log('sub publication', publication);
    if(publication.track){
      attachTrack(div, publication.track);
    }
    publication.on('subscribed', track => attachTrack(div, track));
    publication.on('unsubscribed', track => detachTrack(track));
  }

  function attachTrack(div, track){
    console.log('attachTrack', track);
    div.appendChild(track.attach());
  }

  function trackUnsubscribed(publication) {
    console.log('unsub publication', publication);
    if(publication.track){
      detachTrack(publication.track);
    }
  }
}

function detachTrack(track) {
  console.log('detachTrack', track);
  track.detach().forEach(element => element.remove());
}

function toggleVideo() {
  console.log(videoTrack, room);
  if (videoTrack.isEnabled) {
    videoTrack.disable();
    // room.localParticipant.unpublishTrack(videoTrack);
    console.log('disable');
  } else {
    videoTrack.enable();
    // room.localParticipant.publishTrack(videoTrack);
    console.log('enable');
  }
}

async function switchCamera(){
  facing = !facing;
  // Get new track
  const newTrack = await getVideoTrack();

  // Stop old track
  videoTrack.stop();
  detachTrack(videoTrack);

  // Unpublish previous track
  room.localParticipant.unpublishTrack(videoTrack);

  // Publish new track
  room.localParticipant.publishTrack(newTrack);
}

function getVideoTrack(){
  return Twilio.Video.createLocalVideoTrack({facingMode: facing ? 'user' : {exact: 'environment'}});
}

function leaveMeeting() {
  room.disconnect();
  videoTrack.stop();
  audioTrack.stop();
  detachTrack(videoTrack);
  detachTrack(audioTrack);
}

async function ready() {
  // Note: This allow this sample to run on any Browser
  var cordova = window.cordova;
  if (cordova && cordova.plugins && cordova.plugins.iosrtc) {

    // Expose WebRTC and GetUserMedia SHIM as Globals (Optional)
    // Alternatively WebRTC API will be inside cordova.plugins.iosrtc namespace
    cordova.plugins.iosrtc.registerGlobals();

    // Enable iosrtc debug (Optional)
    cordova.plugins.iosrtc.debug.enable('*', true);
  }
  console.log('loading Twilio');

  await loadScript('https://media.twiliocdn.com/sdk/js/video/releases/2.7.2/twilio-video.js');
  console.log('loaded');
}

if(!window.cordova){
  ready();
}

document.addEventListener('deviceready', ready, false);
</script>
</body>
</html>

Steps to reproduce

Create a cordova project with the attached code and Clicking on Start Meeting then connect to the same room with another twilio client from the desktop.

Expected results

Videos are displayed from remote and local

Actual results

Black video is displayed for the local and no remote videos are showing

Attached is the log file that we got after connecting. We removed the getStats logging to avoid polution

video test.log

samgabriel commented 3 years ago

@hthetiot this is related to #576 not sure if we are doing something wrong but it seems completely broken now

samgabriel commented 3 years ago

iosrtc-6.0.15.txt Incorrect log file was uploaded before. Here is the correct one

hthetiot commented 3 years ago

....

hthetiot commented 3 years ago

"@hthetiot we just tested it from iOS 14 and we can see the video that was previously not displaying"

https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/576

The problem is you said it was working for you, then said "tried to update the WebRTC to M84" without specific information on the branch used task/8.0.0 or 8.0.0-RC1 or 8.0.0-RC2....

And now you come back syaing its not working....

How can we can from "we just tested it from iOS 14 and we can see the video that was previously not displaying" to "no videos are showing from the remote user at all and local videos are not showing on the remote."...

Im not going to maintain Twilio in this conditions; it works for other with Twilio, if you have more Twilio issue i suggest to involve Twilio that you pay for, to get support/fixes.

hthetiot commented 3 years ago

"Using latest 6.0.15 no videos remote videos are showing at ALL" NO

"Using Twillio and 6.0.15 no videos remote videos are showing at ALL" YES

hthetiot commented 3 years ago

@samgabriel I do ot pay for twillio, unless Im provided a api key for testing, and proper test to reproduce, i dont plan to spent more time on this.

hthetiot commented 3 years ago

Please provide/update following information, if we ask for it there is a reason:

Versions affected
Cordova version (e.g 8.0):
Cordova iOS version (e.g 4.6.4):
Plugin version (e.g 6.0.15):
iOS version (e.g 13.3):
Xcode version (e.g 11.3.1):

I doubt that you use cordova 8 and cordova ios 4.6.4 or Xcode 11.3.1, if you do, you may want to update first....

hthetiot commented 3 years ago

"@hthetiot actually after updating the device with iOS 14 to version 6.0.14 the simulator is crashing again so definitely need the M84 version"

No it does not, it work perferctly with 6.0.14 and IOS 14.0.1 as 6.0.15 does.... Not sure what and how you are testing.

hthetiot commented 3 years ago

2020-10-27 14:30:22.502129-0400 Brainfuse[43911:2401989] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=default_D793A74F-0E56-4E32-BE5B-329F1214F1DA already exist\

Its 2020 and Twilio still name all their stream with "default" instead of a clean UUID. What a joke. https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/src/PluginMediaStream.swift#L20

hthetiot commented 3 years ago

To fix this issue, i need an twilio accoundSID and videoToken and authToken. I will not pay twilio to fix an issue for a free open source package. That simple like that. you can send it by email to hthetiot at sylaps.com, if you provide it, i may be able to make a sprint this weekend.

samgabriel commented 3 years ago

@hthetiot I emailed you the API key and secret. please find an email from sgabriel at brainfuse.com

hthetiot commented 3 years ago

@hthetiot I emailed you the API key and secret. please find an email from sgabriel at brainfuse.com

Received, thank you, i will look into it starting Friday night and perform a sprint on 6.0.16 to attempt to fix this last Twillio issue.

samgabriel commented 3 years ago

So I spent sometime checking the logs last night it seems that in the previous versions there would be: iosrtc:MediaStream emitConnected() and then that would fire the publication.on("subscribed") and then call attachTrack above which adds the video.


2020-09-24 12:58:03.570410-0400 Brainfuse[1222:42625] iosrtc:MediaStream emitConnected() +0ms
2020-09-24 12:58:03.571110-0400 Brainfuse[1222:42625] video item on attach helpnow4
2020-09-24 12:58:03.571535-0400 Brainfuse[1222:42625] iosrtc:videoElementsHandler observeVideo() +46s
2020-09-24 12:58:03.571799-0400 Brainfuse[1222:42625] iosrtc:MediaStreamRenderer new() | [element:"[object HTMLVideoElement]"] +45s

But after the latest version we get this emitConnected and then nothing from Twilio.

2020-10-27 14:30:22.587713-0400 Brainfuse[43911:2401989] iosrtc:MediaStream emitConnected() +55ms\
2020-10-27 14:30:22.589834-0400 Brainfuse[43911:2401989] iosrtc:RTCPeerConnection onEvent() | [type:iceconnectionstatechange, data:\{"type":"iceconnectionstatechange","iceConnectionState":"completed"\}] +3ms\
2020-10-27 14:30:22.672252-0400 Brainfuse[43911:2402089] PluginRTCPeerConnection | ondatachannel\
2020-10-27 14:30:22.706928-0400 Brainfuse[43911:2402089] PluginRTCDataChannel#init()\
2020-10-27 14:30:22.707983-0400 Brainfuse[43911:2402089] PluginRTCDataChannel#run()\

IllusionVK commented 3 years ago

Hey @hthetiot I just wanted to say thank you for your work on this. We also are trying to use Twilio with iOSRTC (due to their size and scale they are the best choice for us) and so have been watching this and the previous thread from @samgabriel. I'm not sure how we can help because you clearly have lots of skills in WebRTC area and we don't with native code.. i'm not sure if you have a way for us to donate for bounties etc, or if you have some tasks that I could see if our developers can help with. But again, thanks for your help and time on this, I think it helps a lot of people like us who watch the threads but don't have the skills to contribute 👍👍👍

hthetiot commented 3 years ago

I plan do have a sprint this weekend but anyone that want to comple sample and what the reporter provided to reproduce using Safari debug on device (not emulator) and see where iosRTC shim does not provide or cause Twillio black screen. Some time it's an error that is hidden.

stove654 commented 3 years ago

Hi @hthetiot , thanks you so much for support fix plugin. i give you information bug on twilio

Versions affected

Cordova version: 10
Cordova iOS version: 6.1.0
Plugin version: 6.0.15
iOS version: 14.1
Xcode version: 12.1
WebRTC-adapter version: https://webrtc.github.io/adapter/adapter-latest.js
WebRTC Framework version: twilio video 2.7.2
Link repo: https://github.com/stove654/demo-ios-rtc-twilio

File Index.html


<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8">
        <meta http-equiv="Content-Security-Policy" content="default-src *; style-src 'self' http://* 'unsafe-inline'; script-src 'self' http://* 'unsafe-inline' 'unsafe-eval'" />
        <meta name="format-detection" content="telephone=no">
        <meta name="msapplication-tap-highlight" content="no">
        <meta name="viewport" content="initial-scale=1, width=device-width, viewport-fit=cover">
        <meta name="color-scheme" content="light dark">
        <link rel="stylesheet" href="css/index.css">
        <title>Hello World</title>
    </head>
    <body>
        <div class="app">
            <h1>Apache Cordova</h1>
            <div id="deviceready" class="blink">
                <p class="event listening">Connecting to Device</p>
                <p class="event received">Device is Ready</p>
            </div>
        </div>
        <script src="cordova.js"></script>
        <script src="https://webrtc.github.io/adapter/adapter-latest.js"></script>
        <script src="js/index.js"></script>
    </body>
</html>

File JS

/*
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

// Wait for the deviceready event before using any of Cordova's device APIs.
// See https://cordova.apache.org/docs/en/latest/cordova/events/events.html#deviceready
var isCordova = !document.URL.includes('http');
var twilioVideoUrl = "https://media.twiliocdn.com/sdk/js/video/releases/2.7.2/twilio-video.js";
var tokenTwilioVideoCall = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImN0eSI6InR3aWxpby1mcGE7dj0xIn0.eyJqdGkiOiJTSzg3NTU5NmQ1MDE4ODVmZTk2MTUyNmY5MzUwN2E2NTE2LTE2MDQzNzQwNjQiLCJpc3MiOiJTSzg3NTU5NmQ1MDE4ODVmZTk2MTUyNmY5MzUwN2E2NTE2Iiwic3ViIjoiQUNlMDM5NGRiMWUwOGI1Y2MwODU4NGQzOTJjZmYzM2E2OSIsImV4cCI6MTYwNDM3NzY2NCwiZ3JhbnRzIjp7ImlkZW50aXR5IjoidGVzdGxvaTExMSIsInZpZGVvIjp7InJvb20iOiJ0ZXNsb2kxMDEifX19.fdHnKbDSXmPOG1J9DeZPgyTOXOoJBORES6QJuhqpDPI"
var roomName = 'testloi101'

if (!isCordova) {
    loadScript(twilioVideoUrl, function () {
        _initWebrtc()
    })
} else {
    document.addEventListener('deviceready', onDeviceReady, false);
}

function loadScript(url, cb) {
    var script = document.createElement('script');
    script.onload = function () {
        //do stuff with the script
        cb()
    };
    script.src = url;
    document.head.appendChild(script); //or something of the likes
}

function onDeviceReady() {
    // Cordova is now initialized. Have fun!

    console.log('Running cordova-' + cordova.platformId + '@' + cordova.version);
    document.getElementById('deviceready').classList.add('ready');

    if (window.cordova) {
        window.cordova.plugins.iosrtc.registerGlobals()
        window.cordova.plugins.iosrtc.turnOnSpeaker(true);
    }

    loadScript(twilioVideoUrl, function () {
        _initWebrtc()
    })
}

function _initWebrtc (){
    var Video = Twilio.Video;

    Video.connect(tokenTwilioVideoCall, { name:  roomName}).then(room => {
        console.log('Connected to Room "%s"', room.name);

        room.participants.forEach(participantConnected);
        room.on('participantConnected', participantConnected);

        room.on('participantDisconnected', participantDisconnected);
        room.once('disconnected', error => room.participants.forEach(participantDisconnected));
    });  
}

function participantConnected(participant) {
    console.log('Participant "%s" connected', participant.identity);

    const div = document.createElement('div');
    div.id = participant.sid;
    div.innerText = participant.identity;

    participant.on('trackSubscribed', track => trackSubscribed(div, track));
    participant.on('trackUnsubscribed', trackUnsubscribed);

    participant.tracks.forEach(publication => {
      if (publication.isSubscribed) {
        trackSubscribed(div, publication.track);
      }
    });

    document.body.appendChild(div);
}

function participantDisconnected(participant) {
    console.log('Participant "%s" disconnected', participant.identity);
    document.getElementById(participant.sid).remove();
}

function trackSubscribed(div, track) {
    div.appendChild(track.attach());
}

function trackUnsubscribed(track) {
    track.detach().forEach(element => element.remove());
}

Logs on real device Iphone https://i.imgur.com/A0LZS6t.png

https://i.imgur.com/F7oXigU.png

[Log] Running cordova-ios@6.1.1 (cordova.js, line 1413)
[Log] Connected to Room "tesloi101" (cordova.js, line 1413)
[Warning] 2020-11-03 – "03:54:53.188Z" – "|" – "WARN" – "in" – "[PeerConnectionV2 #1: c331e00f-b70e-42ed-aa59-ca8cadb8c668]:" (cordova.js, line 1413)
"Calling setRemoteDescription with an RTCSessionDescription of type \"answer\" failed with the error \"setRemoteDescription() failed: Failed to set remote answer sdp: The order of m-lines in answer doesn't match order in offer. Rejecting answer.\"."
[Warning] 2020-11-03 – "03:54:53.189Z" – "|" – "WARN" – "in" – "[PeerConnectionV2 #1: c331e00f-b70e-42ed-aa59-ca8cadb8c668]:" (cordova.js, line 1413)
"The SDP was v=0
o=- 3813364493 3813364493 IN IP4 0.0.0.0
s=VMbd7b389128bb626abf6a86144185b116
t=0 0
a=ice-lite
a=msid-semantic: WMS *
a=group:BUNDLE audio video
m=audio 11156 UDP/TLS/RTP/SAVPF 111 0
c=IN IP4 3.235.111.254
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=rtcp:11156 IN IP4 3.235.111.254
a=rtpmap:111 opus/48000/2
a=rtpmap:0 PCMU/8000
a=fmtp:111 minptime=10;useinbandfec=1
a=candidate:2 1 UDP 2013266430 3.235.111.254 11156 typ host
a=candidate:2 2 UDP 2013266429 3.235.111.254 17824 typ host
a=rtcp-mux
a=setup:active
a=mid:audio
a=recvonly
a=ice-ufrag:owwh
a=ice-pwd:l8OdajrSsQoAq0dUtw3N7x
a=fingerprint:sha-256 31:6A:BE:EC:09:19:C2:7B:24:BD:FD:1C:F4:62:DC:22:01:3B:38:AB:54:2E:C9:ED:3C:48:00:04:E0:6F:1C:4E
m=video 11156 UDP/TLS/RTP/SAVPF 127 125 124
c=IN IP4 3.235.111.254
a=rtcp-rsize
a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=rtcp:11156 IN IP4 3.235.111.254
a=rtpmap:127 VP8/90000
a=rtpmap:125 red/90000
a=rtpmap:124 ulpfec/90000
a=rtcp-fb:127 goog-remb
a=rtcp-fb:127 ccm fir
a=rtcp-fb:127 nack
a=rtcp-fb:127 nack pli
a=candidate:2 1 UDP 2013266430 3.235.111.254 11156 typ host
a=candidate:2 2 UDP 2013266429 3.235.111.254 17824 typ host
a=rtcp-mux
a=setup:active
a=mid:video
a=recvonly
a=ice-ufrag:owwh
a=ice-pwd:l8OdajrSsQoAq0dUtw3N7x
a=fingerprint:sha-256 31:6A:BE:EC:09:19:C2:7B:24:BD:FD:1C:F4:62:DC:22:01:3B:38:AB:54:2E:C9:ED:3C:48:00:04:E0:6F:1C:4E
"
[Error] Unhandled Promise Rejection: TwilioError: Client is unable to apply a remote media description
    (anonymous function) (twilio-video.js:24539)
    promiseReactionJob

IllusionVK commented 3 years ago

Hey Harold @hthetiot, I asked Loi (@stove654) to do this work to try and help you. Please let us know if you need other things and we’ll see if we can help

IllusionVK commented 3 years ago

And Harold @hthetiot Loi created a quick repo to help get an app running faster. Not sure if needed but here you go: https://github.com/stove654/demo-ios-rtc-twilio

hthetiot commented 3 years ago

Thank you, the sample in the issue work pretty well.

I did not have time to allocate to this issue, if any one want to give it a try in parallel. The issue is most likely due MediaStreamTrack and possibly Sender/Receiver related. It require to debug to determine the reason of the missing Sender or uncaught error if any.

hthetiot commented 3 years ago

The order of m-lines in answer doesn't match order in offer. Rejecting answer.\"."

This is possibly the reason for the issue. I may have a fix for that.

hthetiot commented 3 years ago

@stove654 thank you for your test, can you try with video only (no audio) to see if the error "The order of m-lines in answer doesn't match order in offer. Rejecting answer" is still present and if you receive video.

IllusionVK commented 3 years ago

@hthetiot FYI, Loi has updated the example app (3-4 minutes after your messages here) so that it gets a twilio token to help with testing

stove654 commented 3 years ago

@hthetiot i updated token from twilio, before it invalid sorry for it. but i try without audio. it return same problem. video not show in screen. can you pull new code on my repo and test again. just open app it will connect to twilio. i tested on

web to web is ok
web to ios: web is ok, ios not show video local and remote
ios to ios: not show local and remote Only problem with twilio.

samgabriel commented 3 years ago

The order of m-lines in answer doesn't match order in offer. Rejecting answer."."

This is possibly the reason for the issue. I may have a fix for that.

@hthetiot we never experienced that issue. I looked at the demo code and they are not forcing the use of plan-b . This issue looks like a unified plan mismatch. @IllusionVK I believe you need to add this line to the Twilio.Video.connect in order for it to work with ioswebrtc


  Twilio.Video.connect(token, {
    tracks: [videoTrack, audioTrack],
    name: roomName,
    sdpSemantics: 'plan-b', // This is the line
    bundlePolicy: 'max-compat'
  })

stove654 commented 3 years ago

The order of m-lines in answer doesn't match order in offer. Rejecting answer."."

This is possibly the reason for the issue. I may have a fix for that.

@hthetiot we never experienced that issue. I looked at the demo code and they are not forcing the use of plan-b . This issue looks like a unified plan mismatch. @IllusionVK I believe you need to add this line to the Twilio.Video.connect in order for it to work with ioswebrtc
  Twilio.Video.connect(token, {
    tracks: [videoTrack, audioTrack],
    name: roomName,
    sdpSemantics: 'plan-b', // This is the line
    bundlePolicy: 'max-compat'
  })

Hi @samgabriel , i updated my code https://github.com/stove654/demo-ios-rtc-twilio. I try test now audio working well but video remote show black screen. i'm not saw any error log in debug safari. On your project, Can it work video remote?

samgabriel commented 3 years ago

The order of m-lines in answer doesn't match order in offer. Rejecting answer."."

This is possibly the reason for the issue. I may have a fix for that.

@hthetiot we never experienced that issue. I looked at the demo code and they are not forcing the use of plan-b . This issue looks like a unified plan mismatch. @IllusionVK I believe you need to add this line to the Twilio.Video.connect in order for it to work with ioswebrtc
  Twilio.Video.connect(token, {
    tracks: [videoTrack, audioTrack],
    name: roomName,
    sdpSemantics: 'plan-b', // This is the line
    bundlePolicy: 'max-compat'
  })
Hi @samgabriel , i updated my code https://github.com/stove654/demo-ios-rtc-twilio. I try test now audio working well but video remote show black screen. i'm not saw any error log in debug safari. On your project, Can it work video remote?

No it is not working now about the 6.0.15 update. It was working with issues on 6.0.14. You can use our fork which is working normally. it is a 6.0.14 plus the #579 and updating WebRTC to M84 https://github.com/Brainfuse/cordova-plugin-iosrtc/releases/tag/7.0.0

samgabriel commented 3 years ago

Hi @hthetiot just wondering if you had a chance to look into this issue?

hthetiot commented 3 years ago

Hi @hthetiot just wondering if you had a chance to look into this issue?

No, I don't have the time for allocating a big debug session for now. Please don't ask again. It will be done when it will be. Unless another developer find the issue.

You may try to debug yourself or ask help from other Twillio user until I have bandwidth to allocate.

hthetiot commented 3 years ago

It was working with issues on 6.0.14. You can use our fork which is working normally. it is a 6.0.14 plus the #579 and updating WebRTC to M84 https://github.com/Brainfuse/cordova-plugin-iosrtc/releases/tag/7.0.0

If so, you just have to found what was changed, I don't think it has anything to do with M84 BTW, also branch 7.0.0 is on M79, and finally from 6.0.14 to 6.0.15 I don't expect changes to be the real reason.

You can also try to apply commits from 6.0.14 to 6.0.15 to understand what changes caused it to break.

hthetiot commented 3 years ago

Notice that https://github.com/Brainfuse/cordova-plugin-iosrtc/releases/tag/7.0.0 does not match 7.0.0 at all, it's still use, and only contains partial 6.0.14.

vs 7.0.0 branch https://github.com/cordova-rtc/cordova-plugin-iosrtc/compare/task/7.0.0...Brainfuse:7.0.0

vs 8.0.0 branch https://github.com/cordova-rtc/cordova-plugin-iosrtc/compare/task/8.0.0...Brainfuse:7.0.0

You cannot use M8x or M7x with that branch since the webRTC implementation changed.

Note: 7.0.x use WebRTC M7x and 8.0.x use m8x

hthetiot commented 3 years ago

Try to upload a diff (txt file) of your work "6.0.14" compare to 6.0.14 actual tag and master, that may help.

hthetiot commented 3 years ago

Also you use M69 WebRTC Headers ... NOT 84... unless its part of the merge commit but i cannot find it. See your own git history: https://github.com/Brainfuse/cordova-plugin-iosrtc/commit/e274983094f7fce2149e624ecdd2f7090dfe8c66#diff-fad31137025bce8fb7bf842213c70f98560c9b163ae153732166b8ba8ce72d7c

hthetiot commented 3 years ago

SEE PR for diff https://github.com/Brainfuse/cordova-plugin-iosrtc/pull/1/files?diff=split&w=1

hthetiot commented 3 years ago

That weird that you miss this fix: https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/d8d9b7bc66482b98279c731a98b75182c3af3640#diff-16cedf80ade01c62bdd1ae931d0492330c0b62bf294c08c095ce2fab21a9298d

https://github.com/cordova-rtc/cordova-plugin-iosrtc/compare/task/8.0.0...Brainfuse:7.0.0#diff-1a65abd52bd140a16020d66ef172e1ba833c7ce0eb3fff54136c75fa62c31ae0R494

return new RTCRtpSender({
        track: track
    });

can you try to apply fixes one by one to see if we can detect what change caused "regression" if regression there is. Also why do you use M84 ? instead of M69 ?

hthetiot commented 3 years ago

I dont understand how you can have m84 or m75 without swift modifications, that should not works:

hthetiot commented 3 years ago

Diff with master vs 7.0.1 fork master.diff.txt

hthetiot commented 3 years ago

pls @samgabriel @stove654 @Brainfuse check previous msgs

hthetiot commented 3 years ago

My issue with missing M84 header in a branch named 7.X reserved for M7X is here:

https://github.com/Brainfuse/cordova-plugin-iosrtc/compare/7.0.1...cordova-rtc:task/8.0.0

Still, if a regression was introduced that made it not working unless using Brainfuse branch then its in there.

samgabriel commented 3 years ago

@hthetiot the tag that works for us is version 7.0.0 master and 7.0.1 was an attempt to use the cordova-ios-rtc master with the M84. 7.0.1 is broken. You are also comparing the current branch task/7.0.0 against our branch. But task/7.0.0 has more commits now.

In any case the issue is not our branch vs your branch. The issue is that 6.0.14 worked but master does not show any videos at all. Trust me I read the code many times over and tried a few debugging sessions but I can not make out what went wrong. In our last debug sessions it seems that the DataTrack events were not being dispatched to Twilio. Not sure if that is the culprit but seem that it could be.

samgabriel commented 3 years ago

@hthetiot regarding the previous comment about the DataTrack. The events are getting dispatched to the web layer. However it seems that none of the twilio handlers are called. So that could mean that twilio never saw the track or the listeners that were added were somehow removed. or may be when clone or replace was called the listeners didn't get copied over. Bottom line is the emit method find an empty array of listeners.

samgabriel commented 3 years ago

@hthetiot also regarding test each commit. some commits didn't have the js file updated like this for example https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/aaf420d72f6168f51665991934a97206376aa9f7 is there an easy one line command to do cordova remove and cordova plugin add which will also repackage the js file? or how would you go about testing each commit?

samgabriel commented 3 years ago

@hthetiot ok so we went and tested the various commits. Commit https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/39d33dadd08cf5f934bb274f71f69a816beefa53 is the last one that works. Next testable commit 830682f6a899538881d9bb7f8222bd5cde9d6514 (previous commits lead to build errors) does not work. Here is the diff https://github.com/cordova-rtc/cordova-plugin-iosrtc/compare/39d33dadd08cf5f934bb274f71f69a816beefa53...830682f6a899538881d9bb7f8222bd5cde9d6514

For our testing we did cordova plugin remove cordova-plugin-iosrtc followed by cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#commit_id

hthetiot commented 3 years ago

@samgabriel

is there an easy one line command to do cordova remove and cordova plugin add which will also repackage the js file?

npm run buildwill compile all js/*.js files to create www/cordova-plugin-iosrtc.js

See https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/package.json#L32

samgabriel commented 3 years ago

@samgabriel

is there an easy one line command to do cordova remove and cordova plugin add which will also repackage the js file?

npm run buildwill compile all js/*.js files to create www/cordova-plugin-iosrtc.js

See https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/package.json#L32

@hthetiot yeah i figured this out but it was not starting. In anycase all the commits in between didn't have a js file change. So it is still a valid test. So the problem happens because of one of these changes https://github.com/cordova-rtc/cordova-plugin-iosrtc/compare/39d33dadd08cf5f934bb274f71f69a816beefa53...830682f6a899538881d9bb7f8222bd5cde9d6514

Let me know if I can get you any more info

samgabriel commented 3 years ago

@hthetiot After writing the previous comment, I was able to find the problem. The issue is this change in https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/830682f6a899538881d9bb7f8222bd5cde9d6514/src/PluginMediaStream.swift#L21-L26 if the code is changed to

if (streamId == nil) {
            // Handle possible duplicate remote trackId with  janus or short duplicate name
            // See: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/432
            if(rtcMediaStream.streamId.count < 36 ){
                             self.id = rtcMediaStream.streamId + "_" + UUID().uuidString;
                        } else {
                            self.id = rtcMediaStream.streamId;
                        }
        } else {
            self.id = streamId!;
        }

And in https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/830682f6a899538881d9bb7f8222bd5cde9d6514/src/PluginMediaStreamTrack.swift#L18-L24

if (trackId == nil) {
            // Handle possible duplicate remote trackId with  janus or short duplicate name
            // See: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/432
            if(rtcMediaStreamTrack.trackId.count<36){
                         self.id = rtcMediaStreamTrack.trackId + "_" + UUID().uuidString;
                        } else {
                          self.id = rtcMediaStreamTrack.trackId;
                        }
        } else {
            self.id = trackId!;
        }

Then it works. We are able to see the remote videos.

There is a problem though with 6.0.15 with the fix no local tracks are being published. And other participants are not able to see them. if we use the https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/830682f6a899538881d9bb7f8222bd5cde9d6514 it shows the local video but 6.0.15 itself does not

We tried this fix with both the https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/830682f6a899538881d9bb7f8222bd5cde9d6514 and the 6.0.15 same fix

hthetiot commented 3 years ago

Thank you @samgabriel after reading some of the diff i suspected it might be the issue. Let me review your comment and make a PR now.

hthetiot commented 3 years ago

you are back on next release 6.0.16 Can you test this PR: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/604

TLDR

Testing

To test bugs/MediaStreamUUID branch

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#bugs/MediaStreamUUID --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

samgabriel commented 3 years ago

@hthetiot we already tested that and it works for incoming tracks but for outgoing tracks they are not showing up on the other side.

hthetiot commented 3 years ago

@hthetiot we already tested that and it works for incoming tracks but for outgoing tracks they are not showing up on the other side.

That may be a separate issue.

samgabriel commented 3 years ago

@hthetiot there is also another issue with the stop method i added the comment to the pull request

cordova-rtc / cordova-plugin-iosrtc

On more time; Using Twillio and 6.0.15 no videos remote videos are showing at ALL #592

Please read first!

Versions affected

Description

Steps to reproduce

Expected results

Actual results

Testing