search

cordova-rtc / cordova-plugin-iosrtc

Cordova iOS plugin exposing the WebRTC W3C API
MIT License
689 stars 339 forks source link

getUserMedia compatiblity with Twilio Video #497

Closed sboudouk closed 4 years ago

sboudouk commented 4 years ago

Expected behavior

Since twilio-video uses getUserMedia to retrieve user's Video & Audio tracks, twilio-video should work as expected.

Observed behavior

TypeError: track must be a LocalAudioTrack, LocalVideoTrack, LocalDataTrack, or MediaStreamTrack

Steps to reproduce the problem

  let globalRegistered = 0;

  const setGlobal = (): void => {
  if (device.platform === 'iOS'){
      if (globalRegistered === 0) {
        console.log('successfully registered global');
        // eslint-disable-next-line
        (cordova as any).plugins.iosrtc.registerGlobals();
        globalRegistered = 1;
      }
    }
  };

  document.addEventListener('deviceready', (): void => {
    setGlobal();
  }, false);

  const handleConnectToRoom = async (): Promise<void> => {
    navigator.getUserMedia({ audio: true, video: true }, (stream): void => {
      console.log('I got a Stream:');
      console.log(stream);
      connect(token.token, {
        name: 'roomname',
        tracks: stream.getTracks(),
      }).then(roomJoined).catch((error): void => {
        console.log('Error while connecting to TwilioRoom: ');
        console.log(error);
      });
    }, (error): void => {
      console.log('Error in getUserMedia ');
      console.log(error);
    });
  };

Platform information

It would be great if that the MediaStreamTrack cake would match with twilio-video since It's a very popular package.

Thanks & keep up the good work.

hthetiot commented 4 years ago

Thank you @sboudouk for this detailed issue. I will see if I can solve the problem.

savgiannis commented 4 years ago

Yeah twilio video is much needed I would appreciate if it is fixed!

hthetiot commented 4 years ago

After some debugging, I found that Twilio is keeping internal reference to MediaStream and MediaStreamTrack, the problem is that if you load TwilioVideo before calling iosrtc registerGlobals the MediaStream and MediaStreamTrack will not refer to the right version of the Prototype. One solution is to Load Cordova, call iosrtc registerGlobals then load TwilioVideo.

<!DOCTYPE HTML>
<html>
    <head>
        <title>
            Twilio Video Room
        </title>
        <meta http-equiv="Content-Security-Policy" content="default-src 'self' data: gap: wss://* https://* 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; media-src *">
        <base href="../">
    </head>
    <body>
        <div id="local-media"></div>
        <div id="remote-media"></div>
        <script type="text/javascript" src="cordova.js"></script>
        <!--<script src="https://media.twiliocdn.com/sdk/js/video/releases/2.0.0/twilio-video.js"></script>-->
        <script type="text/javascript">

        function connect() {

            var result = {
                token: 'YOUR_TWILIO_TOKEN',
                room: 'test'
            };

            // Patch MediaStreamTrack with clone
            MediaStreamTrack.prototype.clone = function () {
                return new MediaStreamTrack(this);
            };

            loadScript('https://media.twiliocdn.com/sdk/js/video/releases/2.0.0/twilio-video.js').then(function () {
                joinRoom(result);
            });
        }

        var scriptUrls = [];
        function loadScript(scriptUrl) {

            if (scriptUrls[scriptUrl]) {
                return Promise.resolve(scriptUrl);
            }

            return new Promise(function(resolve, reject) {
                // load adapter.js
                var script = document.createElement("script");
                script.type = "text/javascript";
                script.src = scriptUrl;
                script.async = false;
                document.getElementsByTagName("head")[0].appendChild(script);
                script.onload = function() {
                    scriptUrls[scriptUrl] = true;
                    console.debug('loadScript.loaded', script.src);
                    resolve(scriptUrl);
                };
            });
        }

        function joinRoom(result) {

            const Video = Twilio.Video;
            Video.createLocalVideoTrack().then(track => {
                const localMediaContainer = document.getElementById('local-media');
                localMediaContainer.appendChild(track.attach());
            });

            Video.connect(result.token, {
                name: result.room
            }).then(room => {
                console.log(`Successfully joined a Room: ${room}`);

                // Attach the Tracks of the Room's Participants.
                var remoteMediaContainer = document.getElementById('remote-media');
                    room.participants.forEach(function(participant) {
                    console.log("Already in Room: '" + participant.identity + "'");
                    participantConnected(participant, remoteMediaContainer);
                });

                room.on('participantConnected', participant => {
                    console.log(`A remote Participant connected: ${participant}`);
                    participantConnected(participant);
                });

                room.on('participantDisconnected', participant => {
                    console.log(`A remote Participant connected: ${participant}`);
                    participantDisconnected(participant);
                });

            }, error => {
                console.error(`Unable to connect to Room: ${error.message}`);
            });

            function participantConnected(participant) {
                console.log('Participant "%s" connected', participant.identity);
                const div = document.createElement('div');
                div.id = participant.sid;
                participant.on('trackSubscribed', track => trackSubscribed(div, track));
                participant.on('trackUnsubscribed', trackUnsubscribed);
                participant.tracks.forEach(publication => {
                    if (publication.isSubscribed) {
                        trackSubscribed(div, publication.track);
                    }
                });

                document.getElementById('remote-media').appendChild(div);
            }

            function participantDisconnected(participant) {
                console.log('Participant "%s" disconnected', participant.identity);

                var div = document.getElementById(participant.sid)
                if (div) {
                    div.remove();
                }
            }

            function trackSubscribed(div, track) {
                div.appendChild(track.attach());
            }

            function trackUnsubscribed(track) {
                track.detach().forEach(element => element.remove());
            }
        }

        document.addEventListener('deviceready', function () {
            // Note: This allow this sample to run on any Browser
            var cordova = window.cordova;
            if (cordova && cordova.plugins && cordova.plugins.iosrtc) {

                // Expose WebRTC and GetUserMedia SHIM as Globals (Optional)
                // Alternatively WebRTC API will be inside cordova.plugins.iosrtc namespace
                cordova.plugins.iosrtc.registerGlobals();

                // Enable iosrtc debug (Optional)
                cordova.plugins.iosrtc.debug.enable('*', true);
            }

            connect();
        }, false);
        </script>
    </body>
</html>

But even with loading TwilioVideo after iosrtc registerGlobals it look like there is another issue with Twilio SDP.

Screen Shot 2020-04-26 at 3 44 37 PM
hthetiot commented 4 years ago

@savgiannis

I would appreciate if it is fixed!

It's not an issue with iosrtc but an issue with the way TwilioVideo is written. Still I'm looking into possible workarround.

hthetiot commented 4 years ago

To solve the error "The order of m-lines in answer doesn't match order in offer. Rejecting answer" cause by twilio non detection of plan-b destpite iosrtc supporting unifed-plan.

Options sdpSemantics: 'plan-b' and bundlePolicy: 'max-compat' are required on Twilio.Video.connect:

Twilio.Video.connect(result.token, {
    name: result.room,
    sdpSemantics: 'plan-b',
    bundlePolicy: 'max-compat'
})

See Updated sample:

<!DOCTYPE HTML>
<html>
    <head>
        <title>
            Twilio Video Room
        </title>
        <meta http-equiv="Content-Security-Policy" content="default-src 'self' data: gap: wss://* https://* 'unsafe-eval' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; media-src *">
        <base href="../">
    </head>
    <body>
        <div id="local-media"></div>
        <div id="remote-media"></div>
        <script type="text/javascript" src="cordova.js"></script>
        <!--<script src="https://media.twiliocdn.com/sdk/js/video/releases/2.0.0/twilio-video.js"></script>-->
        <script type="text/javascript">

        function connect() {

            var result = {
                token: 'YOUR_TWILIO_TOKEN',
                room: 'test'
            };

            // Patch MediaStreamTrack with clone
            MediaStreamTrack.prototype.clone = function () {
                return this;
            };

            loadScript('https://media.twiliocdn.com/sdk/js/video/releases/2.0.0/twilio-video.js').then(function () {
                joinRoom(result);
            });
        }

        var scriptUrls = [];
        function loadScript(scriptUrl) {

            if (scriptUrls[scriptUrl]) {
                return Promise.resolve(scriptUrl);
            }

            return new Promise(function(resolve, reject) {
                // load adapter.js
                var script = document.createElement("script");
                script.type = "text/javascript";
                script.src = scriptUrl;
                script.async = false;
                document.getElementsByTagName("head")[0].appendChild(script);
                script.onload = function() {
                    scriptUrls[scriptUrl] = true;
                    console.debug('loadScript.loaded', script.src);
                    resolve(scriptUrl);
                };
            });
        }

        function joinRoom(result) {

            const Video = Twilio.Video;
            Video.createLocalVideoTrack().then(track => {
                const localMediaContainer = document.getElementById('local-media');
                localMediaContainer.appendChild(track.attach());
            });

            Video.connect(result.token, {
                name: result.room,
                sdpSemantics: 'plan-b',
                bundlePolicy: 'max-compat'
            }).then(room => {
                console.log(`Successfully joined a Room: ${room}`);

                // Attach the Tracks of the Room's Participants.
                var remoteMediaContainer = document.getElementById('remote-media');
                    room.participants.forEach(function(participant) {
                    console.log("Already in Room: '" + participant.identity + "'");
                    participantConnected(participant, remoteMediaContainer);
                });

                room.on('participantConnected', participant => {
                    console.log(`A remote Participant connected: ${participant}`);
                    participantConnected(participant);
                });

                room.on('participantDisconnected', participant => {
                    console.log(`A remote Participant connected: ${participant}`);
                    participantDisconnected(participant);
                });

            }, error => {
                console.error(`Unable to connect to Room: ${error.message}`);
            });

            function participantConnected(participant) {
                console.log('Participant "%s" connected', participant.identity);
                const div = document.createElement('div');
                div.id = participant.sid;
                participant.on('trackSubscribed', (track) => {
                    trackSubscribed(div, track)
                });
                participant.on('trackUnsubscribed', trackUnsubscribed);
                participant.tracks.forEach(publication => {
                    if (publication.isSubscribed) {
                        trackSubscribed(div, publication.track);
                    }
                });

                document.getElementById('remote-media').appendChild(div);
            }

            function participantDisconnected(participant) {
                console.log('Participant "%s" disconnected', participant.identity);

                var div = document.getElementById(participant.sid)
                if (div) {
                    div.remove();
                }
            }

            function trackSubscribed(div, track) {
                div.appendChild(track.attach());
            }

            function trackUnsubscribed(track) {
                track.detach().forEach(element => element.remove());
            }
        }

        document.addEventListener('deviceready', function () {
            // Note: This allow this sample to run on any Browser
            var cordova = window.cordova;
            if (cordova && cordova.plugins && cordova.plugins.iosrtc) {

                // Expose WebRTC and GetUserMedia SHIM as Globals (Optional)
                // Alternatively WebRTC API will be inside cordova.plugins.iosrtc namespace
                cordova.plugins.iosrtc.registerGlobals();

                // Enable iosrtc debug (Optional)
                cordova.plugins.iosrtc.debug.enable('*', true);
            }

            connect();
        }, false);
        </script>
    </body>
</html>

This result in video been received on browser from iosrtc but not on iosRTC from browser, more debugging is required to understand why trackSubscribed is not triggered on iosRTC using TwilioVideo

hthetiot commented 4 years ago

I dont want to spent to much Twilio credits, if you want me to debug more I will need that someone give me access to a twilio accoundSID and videoToken and videoSecret.

sboudouk commented 4 years ago

I got you @hthetiot I'll give you mine by mail.

hthetiot commented 4 years ago

Thank you @sboudouk

hthetiot commented 4 years ago

I need to release 6.0.12 before but I have found some way to make it works. (6.0.12 https://github.com/cordova-rtc/cordova-plugin-iosrtc/milestone/25)

This may be on 6.0.1 because I had to SHAM MediaStreamTrack better and also override existing MediaStreamTrack and MediaStream prototype (I know it's dirty) to allow loading Twlio before iosrtc and still use iosRTC shim, this may cause issue with webrtc-adapter i need more testing.

Maxouhell commented 4 years ago

Hello ! We are currently facing the same problem.

Do you have any news on that ?

Thanks !

hthetiot commented 4 years ago

No update yet, the fix will be a combination of https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-619563313 and #508

Also IF there were update you would see update here. If you want you can also contribute that free software, therefore it's done only on my spare time, I'm not your employee or at your service.

Maxouhell commented 4 years ago

Seriously ? Oo Calm down, I'm just asking if you found a solution while i'm searching for it...

I would be very happy to contribute if i find the solution... ><'

hthetiot commented 4 years ago

I'm just asking if you found a solution while i'm searching for it...

-> https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-619563313

cj commented 4 years ago

@hthetiot

First off thank you for this amazing plugin!

I just tried the fix you suggest but I got getUserMedia is not supported:

Make a symbolic breakpoint at UIViewAlertForUnsatisfiableConstraints to catch this in the debugger.
The methods in the UIConstraintBasedLayoutDebugging category on UIView listed in <UIKitCore/UIView.h> may also be helpful.
2020-05-18 19:23:32.188210-0500 ***[2292:686887] [Snapshotting] Snapshotting a view (0x10457b560, _UIReplicantView) that has not been rendered at least once requires afterScreenUpdates:YES.
2020-05-18 19:23:33.074137-0500 *[2292:686887] WARN: 2020-05-19 00:23:33.071Z | WARN in [createLocalTracks #1]: Call to getUserMedia failed: Error: getUserMedia is not supported
2020-05-18 19:23:33.079292-0500 *[2292:686887] WARN: 2020-05-19 00:23:33.074Z | WARN in [createLocalTracks #2]: Call to getUserMedia failed: Error: getUserMedia is not supported
2020-05-18 19:23:33.079442-0500 *[2292:686887] ERROR: Unable to connect to Room: getUserMedia is not supported

I can give you access to my Twilio account accoundSID and videoToken and videoSecret just shoot me an email cjlazell at gmail.com. If you add a Github sponsor button, I'd love to sponsor this project too! https://github.com/sponsors

Cheers!

cj commented 4 years ago

@hthetiot wups the getUserMedia is not supported was just because I forgot to initialize it (sorry long day). It's now loading, but there seems to be another issue, sound is fine, but the video is just black.

cj commented 4 years ago

ok - figured out the black video, this was the cause https://github.com/twilio/video-quickstart-js/issues/116#issuecomment-614872845. Now the local participant shows the video (the cordova app running on ios)!

There is another issue. If i now open a browser on a computer, I can see and hear the iOS video, but on the device itself I can't see any audio or video from the computer. If I try it on iOS using the safari browser instead of the cordova app, everything works just fine. I see the iOS local video and the connected computers audio and video.

cj commented 4 years ago

iOS cordova is attaching the audio (I can hear it) of the computer, but just not the video.

cj commented 4 years ago

So I think I may have found the reason why the computer video does not show up in iOS cordova when someone joins. https://gist.github.com/cj/3226ecd8a4acdf616d66360565967dd3#file-gistfile1-txt-L33

ERROR: pluginMediaStream with id=default_63D585DA-B0D2-44FF-A486-058AB001694C already exist

Seems like https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/src/PluginMediaStream.swift#L24 might not be working correctly.

Doing something hacky like self.id = rtcMediaStream.streamId + "_" + UUID().uuidString + String(rtcMediaStream.streamId.count); gets ride of the error, but the video still never gets displayed.

hthetiot commented 4 years ago

Thank you @cj im working on it in #508 keep looking you might find something else. I will also look at your workaround for MediaStreamId.

hthetiot commented 4 years ago

508 is now ready and should fix the issue with Twilio.

@cj @sboudouk give it a try using the sample here https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-619563313 and the branch bugs/ontrack (see install instruction on #508 description).

cj commented 4 years ago

@hthetiot no problem - I Just gave https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/508 a try and it worked perfectly, even with the Twilio NPM package! Thank you!

You should 100% add a sponsor badge, and I'm sure I'm not the only one that would use it to support your fantastic plugin and hard work!

hthetiot commented 4 years ago

Thank you @cj, I have the sponsor I need to finalize the GitHub request.

@sboudouk your fix is ready see last comment.

hthetiot commented 4 years ago

@cj I made some change to try to handle #505 issue can you test #508 again with you have a moment:

Long story short, I was creating a new MediaStream everytime with possibly one track instead of 2 and it cause issue for Library that still use event.streams[0] on track event instead of creating there own MediaStream like that should. See https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/505#issuecomment-631402350

Thank you for your help.

cj commented 4 years ago

@hthetiot that latest version did not work, the computer video never got added to iOS.

2020-05-20 10:58:52.372471-0500 ***[3248:973618] iosrtc:MediaStreamRenderer refresh() +0ms
2020-05-20 10:58:52.372771-0500 ***[3248:973618] iosrtc:MediaStreamRenderer refresh() | no video track yet +0ms
2020-05-20 10:58:52.373227-0500 ***[3248:973618] iosrtc:MediaStreamRenderer refresh() | [data:{"elementLeft":0,"elementTop":268,"elementWidth":400,"elementHeight":200,"videoViewWidth":400,"videoViewHeight":200,"visible":true,"opacity":1,"zIndex":0,"mirrored":false,"objectFit":"contain","clip":true,"borderRadius":0}] +0ms
2020-05-20 10:58:52.373496-0500 ***[3248:973618] iosrtcPlugin#MediaStreamRenderer_refresh()
2020-05-20 10:58:52.373634-0500 ***[3248:973618] PluginMediaStreamRenderer#refresh() [elementLeft:0.0, elementTop:268.0, elementWidth:400.0, elementHeight:200.0, videoViewWidth:400.0, videoViewHeight:200.0, visible:true, opacity:1.0, zIndex:0.0, mirrored:false, clip:true, borderRadius:0.0]
2020-05-20 10:58:52.373946-0500 ***[3248:973618] iosrtc:MediaStreamRenderer render() [stream:{"_listeners":{},"_id":"2013180324-2645161682-1254368535-3708531460","_active":true,"connected":true,"_audioTracks":{},"_videoTracks":{"690220b0-c3b5-455a-9a71-2b6ac0224faf":{"_listeners":{"ended":[null,null,null,null]},"id":"690220b0-c3b5-455a-9a71-2b6ac0224faf","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}},"_blobId":"MediaStream_2013180324-2645161682-1254368535-3708531460"}] +0ms
2020-05-20 10:58:52.374165-0500 ***[3248:973618] iosrtcPlugin#MediaStreamRenderer_render()
2020-05-20 10:58:52.374331-0500 ***[3248:973618] PluginMediaStreamRenderer#render()
2020-05-20 10:58:52.374835-0500 ***[3248:973618] iosrtc:RTCPeerConnection onEvent() | [type:addstream, data:{"stream":{"id":"default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1","audioTracks":{"75471f2e-27b6-48e5-a392-529d3d8c1603":{"readyState":"ended","id":"75471f2e-27b6-48e5-a392-529d3d8c1603","kind":"audio","enabled":true,"trackId":"75471f2e-27b6-48e5-a392-529d3d8c1603"}},"videoTracks":{}},"type":"addstream","streamId":"default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1"}] +38ms
2020-05-20 10:58:52.374947-0500 ***[3248:973618] iosrtc:MediaStream create() | [dataFromEvent:{"id":"default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1","audioTracks":{"75471f2e-27b6-48e5-a392-529d3d8c1603":{"readyState":"ended","id":"75471f2e-27b6-48e5-a392-529d3d8c1603","kind":"audio","enabled":true,"trackId":"75471f2e-27b6-48e5-a392-529d3d8c1603"}},"videoTracks":{}}] +31ms
2020-05-20 10:58:52.375105-0500 ***[3248:973618] iosrtc:MediaStream new MediaStream(arg) | [arg:[]] +0ms
2020-05-20 10:58:52.375423-0500 ***[3248:973618] iosrtcPlugin#MediaStream_init()
2020-05-20 10:58:52.375478-0500 ***[3248:973618] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=default_8F7023E5-2983-4A47-B6AF-BCD3EB801FF1 already exist
2020-05-20 10:58:52.375661-0500 ***[3248:973618] iosrtcPlugin#MediaStream_setListener()
2020-05-20 10:58:52.375780-0500 ***[3248:973618] iosrtc:MediaStreamTrack new() | [dataFromEvent:{"readyState":"ended","id":"75471f2e-27b6-48e5-a392-529d3d8c1603","kind":"audio","enabled":true,"trackId":"75471f2e-27b6-48e5-a392-529d3d8c1603"}] +38ms
2020-05-20 10:58:52.375809-0500 ***[3248:973786] PluginMediaStream#setListener()
2020-05-20 10:58:52.375878-0500 ***[3248:973618] iosrtcPlugin#MediaStreamTrack_setListener()
2020-05-20 10:58:52.375968-0500 ***[3248:973785] PluginMediaStreamTrack#setListener() [kind:audio, id:75471f2e-27b6-48e5-a392-529d3d8c1603]
2020-05-20 10:58:52.375981-0500 ***[3248:973618] Error in Success callbackId: iosrtcPlugin947657497 : TypeError: undefined is not an object (evaluating 'stream.emitConnected')
2020-05-20 10:58:52.376675-0500 ***[3248:973618] onEvent@ionic://app/plugins/cordova-plugin-iosrtc/dist/cordova-plugin-iosrtc.js:2428:11
cj commented 4 years ago

@hthetiot I just re-installed this commit aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600 and it does indeed work. So something in this new one is breaking it.

hthetiot commented 4 years ago

2020-05-20 10:58:52.375981-0500 ***[3248:973618] Error in Success callbackId: iosrtcPlugin947657497 : TypeError: undefined is not an object (evaluating 'stream.emitConnected')

hthetiot commented 4 years ago

Sorry about that the lint failed to catch the issue. Can you try 44e5b8b, thank you again for your help. I did not have this issue in the sample I think because I used ontrack only.

hthetiot commented 4 years ago

@cj How did you use the package because you need to load twilio after iosrtc we just d'like to know me and @sboudouk

hthetiot commented 4 years ago

@cj if you have a working Twilio-video.js we would be curious to see how. See https://github.com/twilio/twilio-video.js/issues/963#issuecomment-634203022 details.

hthetiot commented 4 years ago

Updated Twilio possible issue with this commit:

cj commented 4 years ago

@hthetiot @sboudouk This was just me playing around, but it works https://gist.github.com/cj/2beabb496fde34da009c946775be6021 you basically just require the twilio package after you have registered iosrtc.

p.s. I was using "twilio-video": "^2.4.0"

sboudouk commented 4 years ago

@cj Using this code, did you manage to get remoteParticipant track to display on iOS device ?

Thanks for sharing.

cj commented 4 years ago

@sboudouk yes - it was working perfectly in iOS!

hthetiot commented 4 years ago

Thank you @cj

Closing the issue for now then.

matijagrcic commented 4 years ago

Are above changes available in 6.0.12 https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/425#issuecomment-635225234 ? Only asking as I'm getting same error as https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/497#issuecomment-630579714

ERROR: pluginMediaStream with id=default_63D585DA-B0D2-44FF-A486-058AB001694C already exist

Thanks for this whole thread, helped greatly so far.

sboudouk commented 4 years ago

@matijagrcic try to use https://github.com/cordova-rtc/cordova-plugin-iosrtc/commit/aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600 commit, it works for me, when 6.0.12 don't, you should give it a try.

matijagrcic commented 4 years ago

Thanks, how would i install this this commit?

sboudouk commented 4 years ago 

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc\#aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600 --save
cordova platform remove ios --no-save
cordova platform add ios --no-save
matijagrcic commented 4 years ago

Thanks, wasn't sure do i just reference some branch like mentioned here https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/508 or the commit like you did.

sboudouk commented 4 years ago

@matijagrcic try to use aee2b39 commit

matijagrcic commented 4 years ago

Yep, all good, already trying that now.

matijagrcic commented 4 years ago

I'm still seeing the following

2020-06-03 23:10:38.614782+0100 App[43033:442187] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist

full log below:

2020-06-03 23:10:38.606955+0100 App[43033:442187] iosrtc registerGlobals() +0ms
2020-06-03 23:10:38.607696+0100 App[43033:442187] iosrtc restoreCallbacksSupport() +0ms
2020-06-03 23:10:38.608673+0100 App[43033:442187] iosrtc:getUserMedia [original constraints:{"audio":false,"video":{"facingMode":"user"}}] +1m
2020-06-03 23:10:38.609096+0100 App[43033:442187] iosrtc:getUserMedia [computed constraints:{"video":{"facingMode":{"exact":"user"}}}] +0ms
2020-06-03 23:10:38.611019+0100 App[43033:442187] iosrtcPlugin#getUserMedia()
2020-06-03 23:10:38.611164+0100 App[43033:442187] PluginGetUserMedia#call()
2020-06-03 23:10:38.611266+0100 App[43033:442187] PluginGetUserMedia#call() | video authorization: authorized
2020-06-03 23:10:38.611432+0100 App[43033:442187] PluginGetUserMedia#call() | video requested
2020-06-03 23:10:38.611609+0100 App[43033:442187] PluginGetUserMedia#call() | chosen video constraints: {
    facingMode =     {
        exact = user;
    };
}
2020-06-03 23:10:38.612396+0100 App[43033:442187] PluginMediaStream#init()
2020-06-03 23:10:38.612649+0100 App[43033:442187] PluginMediaStreamTrack#init()
2020-06-03 23:10:38.612805+0100 App[43033:442187] PluginMediaStreamTrack#run() [kind:video, id:48E7AD91-13D7-4E37-82D5-3136A8D0A06F]
2020-06-03 23:10:38.612883+0100 App[43033:442187] PluginMediaStream#run()
2020-06-03 23:10:38.613912+0100 App[43033:442187] iosrtc:getUserMedia getUserMedia() | success +7ms
2020-06-03 23:10:38.614085+0100 App[43033:442187] iosrtc:MediaStream create() | [dataFromEvent:{"id":"13975956-A8BA-4D7D-ACB6-2FF25624A479","audioTracks":{},"videoTracks":{"48E7AD91-13D7-4E37-82D5-3136A8D0A06F":{"readyState":"live","id":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F","kind":"video","enabled":true,"trackId":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F"}}}] +1m
2020-06-03 23:10:38.614224+0100 App[43033:442187] iosrtc:MediaStream new MediaStream(arg) | [arg:[]] +0ms
2020-06-03 23:10:38.614673+0100 App[43033:442187] iosrtcPlugin#MediaStream_init()
2020-06-03 23:10:38.614782+0100 App[43033:442187] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist
2020-06-03 23:10:38.614914+0100 App[43033:442187] iosrtcPlugin#MediaStream_setListener()
2020-06-03 23:10:38.615018+0100 App[43033:443387] PluginMediaStream#setListener()
2020-06-03 23:10:38.615068+0100 App[43033:442187] iosrtc:MediaStreamTrack new() | [dataFromEvent:{"readyState":"live","id":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F","kind":"video","enabled":true,"trackId":"48E7AD91-13D7-4E37-82D5-3136A8D0A06F"}] +1m
2020-06-03 23:10:38.615697+0100 App[43033:442187] iosrtcPlugin#MediaStreamTrack_setListener()
2020-06-03 23:10:38.615828+0100 App[43033:443849] PluginMediaStreamTrack#setListener() [kind:video, id:48E7AD91-13D7-4E37-82D5-3136A8D0A06F]
2020-06-03 23:10:38.615876+0100 App[43033:442187] iosrtc:MediaStream emitConnected() +1ms
2020-06-03 23:10:38.615998+0100 App[43033:442187] iosrtc:MediaStream getAudioTracks() +0ms
2020-06-03 23:10:38.616121+0100 App[43033:442187] iosrtc:MediaStream getVideoTracks() +0ms
// require Twilio after we have registered iosrtc
const Video = require("twilio-video");

Video.createLocalVideoTrack({ facingMode: "user" }).then((track) => {
  this.localPreviewVideoElement.nativeElement.appendChild(track.attach());
});

Any pointers as it seems to get the video properly but then fails?

sboudouk commented 4 years ago

@matijagrcic can you provide full code please ? With Twilio and iOS version ?

matijagrcic commented 4 years ago

All info below, anything else needed please let me know.

❯ npm view twilio-video version
2.5.0

iOS version: 12+

export enum CameraDirection {
  Front,
  Rear,
}
type VideoConfiguration = {
  [key in CameraDirection]: string;
};

export const VideoConfigurations: VideoConfiguration = {
  [CameraDirection.Front]: 'user',
  [CameraDirection.Rear]: 'environment',
};

function registerWebRTCForiOS() {
  var cordova = window.cordova;
  if (cordova && cordova.plugins && cordova.plugins.iosrtc) {
    cordova.plugins.iosrtc.registerGlobals();

    // patch MediaStreamTrack with clone
    MediaStreamTrack.prototype.clone = function () {
      return this;
    };
    cordova.plugins.iosrtc.debug.enable('*', true);
  }
}

function showLocalVideoPreview(cameraDirection: CameraDirection) {
  registerWebRTCForiOS();

  // we need to require Twilio after we have registered iosrtc
  const Video = require('twilio-video');

  Video.createLocalVideoTrack({
    facingMode: VideoConfigurations[cameraDirection],
  }).then((track) => {
    this.localPreviewVideoElement.nativeElement.appendChild(track.attach());
  });
}

showLocalVideoPreview(CameraDirection.Front);
sboudouk commented 4 years ago

You should make sure that it's iosrtc MediaStreamTrack that is used before calling Video, maybe in the React's useEffect function:

 useEffect(() => {
    const cordova = window.cordova

    if (cordova && cordova.plugins && cordova.plugins.iosrtc) {

      // Expose WebRTC and GetUserMedia SHIM as Globals (Optional)
      // Alternatively WebRTC API will be inside cordova.plugins.iosrtc namespace
      cordova.plugins.iosrtc.registerGlobals()

      // Patch MediaStreamTrack with clone
      MediaStreamTrack.prototype.clone = function () {
        return this
      }

      // Enable iosrtc debug (Optional)
      cordova.plugins.iosrtc.debug.enable('*', false)
    }
    MediaStreamTrack.prototype.clone = function () {
      return this
    }
    connect(name, room)
  // eslint-disable-next-line
  }, [])

Also, while running on device, put a breakpoint before calling "Video.createLocalVideoTrack" and in Safari's console, you can try to log MediaStreamTrack.

Even if I think that's it's not the issue, because you would get a different error, so it might be an issue with iOS 12. Can you test this on iOS 13+ ?

matijagrcic commented 4 years ago

Thank you @sboudouk. Yeah it's strange, as per the output log all seems fine (stream is acquired) but then it errors out and so the output isn't displayed. Will try running it under iOS 13+ and will follow up with the result but this might not be feasible from the business perspective.

matijagrcic commented 4 years ago

I've managed to get it working on iOS 12+ (no getting ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist anymore) by changing the code slightly but now it's complaining about the 2020-06-05 13:09:44.361364+0100 App[12577:99329] [LayoutConstraints] Unable to simultaneously satisfy constraints..

I've added full log below, based on searching thru this repo I've tried setting width/height but I've also tried without them.

Reference: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/375#issuecomment-546917118

Would you have any clue of what can be wrong here?

020-06-05 13:09:44.344110+0100 App[12577:99329] iosrtc:getUserMedia [original constraints:{"audio":false,"video":{"facingMode":"user","width":640,"height":480}}] +20s
2020-06-05 13:09:44.344261+0100 App[12577:99329] iosrtc:getUserMedia [computed constraints:{"video":{"width":{"exact":640},"height":{"exact":480},"facingMode":{"exact":"user"}}}] +0ms
2020-06-05 13:09:44.344456+0100 App[12577:99329] iosrtcPlugin#getUserMedia()
2020-06-05 13:09:44.344547+0100 App[12577:99329] PluginGetUserMedia#call()
2020-06-05 13:09:44.344655+0100 App[12577:99329] PluginGetUserMedia#call() | video authorization: authorized
2020-06-05 13:09:44.344872+0100 App[12577:99329] PluginGetUserMedia#call() | video requested
2020-06-05 13:09:44.345765+0100 App[12577:99329] PluginGetUserMedia#call() | chosen video constraints: {
    facingMode =     {
        exact = user;
    };
    height =     {
        exact = 480;
    };
    width =     {
        exact = 640;
    };
}
2020-06-05 13:09:44.345961+0100 App[12577:99329] PluginMediaStream#init()
2020-06-05 13:09:44.346278+0100 App[12577:99329] PluginMediaStreamTrack#init()
2020-06-05 13:09:44.346433+0100 App[12577:99329] PluginMediaStreamTrack#run() [kind:video, id:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.346516+0100 App[12577:99329] PluginMediaStream#run()
2020-06-05 13:09:44.347468+0100 App[12577:99329] iosrtc:getUserMedia getUserMedia() | success +5ms
2020-06-05 13:09:44.347642+0100 App[12577:99329] iosrtc:MediaStream create() | [dataFromEvent:{"id":"C390DCA1-2882-45D2-A07F-C8F397F769A2","audioTracks":{},"videoTracks":{"DD36969E-8111-4070-811F-FCF5CFC7A1B5":{"readyState":"live","id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","enabled":true,"trackId":"DD36969E-8111-4070-811F-FCF5CFC7A1B5"}}}] +19s
2020-06-05 13:09:44.347922+0100 App[12577:99329] iosrtc:MediaStream new MediaStream(arg) | [arg:[]] +0ms
2020-06-05 13:09:44.348081+0100 App[12577:99329] iosrtcPlugin#MediaStream_init()
2020-06-05 13:09:44.348179+0100 App[12577:99329] iosrtcPlugin#MediaStream_init() | ERROR: pluginMediaStream with id=C390DCA1-2882-45D2-A07F-C8F397F769A2 already exist
2020-06-05 13:09:44.348311+0100 App[12577:99329] iosrtcPlugin#MediaStream_setListener()
2020-06-05 13:09:44.348415+0100 App[12577:100344] PluginMediaStream#setListener()
2020-06-05 13:09:44.348994+0100 App[12577:99329] iosrtc:MediaStreamTrack new() | [dataFromEvent:{"readyState":"live","id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","enabled":true,"trackId":"DD36969E-8111-4070-811F-FCF5CFC7A1B5"}] +19s
2020-06-05 13:09:44.349178+0100 App[12577:99329] iosrtcPlugin#MediaStreamTrack_setListener()
2020-06-05 13:09:44.349303+0100 App[12577:100344] PluginMediaStreamTrack#setListener() [kind:video, id:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.349340+0100 App[12577:99329] iosrtc:MediaStream emitConnected() +1ms
2020-06-05 13:09:44.350208+0100 App[12577:99329] iosrtc:MediaStream getAudioTracks() +0ms
2020-06-05 13:09:44.350381+0100 App[12577:99329] iosrtc:MediaStream getVideoTracks() +2ms
2020-06-05 13:09:44.352594+0100 App[12577:99329] iosrtc:MediaStream new MediaStream(arg) | [arg:] +2ms
2020-06-05 13:09:44.352838+0100 App[12577:99329] iosrtcPlugin#MediaStream_init()
2020-06-05 13:09:44.353378+0100 App[12577:99329] PluginMediaStream#init()
2020-06-05 13:09:44.353612+0100 App[12577:99329] PluginMediaStream#run()
2020-06-05 13:09:44.353787+0100 App[12577:99329] iosrtcPlugin#MediaStream_setListener()
2020-06-05 13:09:44.353906+0100 App[12577:100344] PluginMediaStream#setListener()
2020-06-05 13:09:44.353945+0100 App[12577:99329] iosrtc:MediaStream getVideoTracks() +0ms
2020-06-05 13:09:44.354068+0100 App[12577:99329] iosrtc:MediaStream addTrack() [track:{"_listeners":{"ended":[null,null]},"id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}] +0ms
2020-06-05 13:09:44.354189+0100 App[12577:99329] iosrtcPlugin#MediaStream_addTrack()
2020-06-05 13:09:44.354294+0100 App[12577:100344] PluginMediaStream#addTrack()
2020-06-05 13:09:44.354328+0100 App[12577:99329] iosrtc:MediaStream emitConnected() +0ms
2020-06-05 13:09:44.354602+0100 App[12577:99329] AppHealthDEBUG: createLocalVideoTrack
2020-06-05 13:09:44.354862+0100 App[12577:100344] PluginMediaStream#addTrack() | video track added
2020-06-05 13:09:44.355081+0100 App[12577:99329] iosrtc:MediaStream new MediaStream(arg) | [arg:] +1ms
2020-06-05 13:09:44.355187+0100 App[12577:100344] PluginMediaStream | OnAddTrack [label:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.355516+0100 App[12577:100344] PluginMediaStreamTrack#run() [kind:video, id:DD36969E-8111-4070-811F-FCF5CFC7A1B5]
2020-06-05 13:09:44.355691+0100 App[12577:99329] iosrtcPlugin#MediaStream_init()
2020-06-05 13:09:44.356580+0100 App[12577:99329] PluginMediaStream#init()
2020-06-05 13:09:44.357129+0100 App[12577:99329] PluginMediaStream#run()
2020-06-05 13:09:44.357344+0100 App[12577:99329] iosrtcPlugin#MediaStream_setListener()
2020-06-05 13:09:44.357507+0100 App[12577:99329] iosrtc:MediaStream getVideoTracks() +1ms
2020-06-05 13:09:44.357645+0100 App[12577:99329] iosrtc:MediaStream addTrack() [track:{"_listeners":{"ended":[null,null,null]},"id":"DD36969E-8111-4070-811F-FCF5CFC7A1B5","kind":"video","muted":false,"readyState":"live","_enabled":true,"_ended":false}] +0ms
2020-06-05 13:09:44.357922+0100 App[12577:99329] iosrtcPlugin#MediaStream_addTrack()
2020-06-05 13:09:44.358157+0100 App[12577:99329] iosrtc:MediaStream emitConnected() +0ms
2020-06-05 13:09:44.358937+0100 App[12577:99329] iosrtc:videoElementsHandler new video element added +13s
2020-06-05 13:09:44.359205+0100 App[12577:99329] iosrtc:videoElementsHandler observeVideo() +0ms
2020-06-05 13:09:44.359466+0100 App[12577:99329] iosrtc:MediaStreamRenderer new() | [element:"[object HTMLVideoElement]"] +13s
2020-06-05 13:09:44.359672+0100 App[12577:99329] iosrtcPlugin#new_MediaStreamRenderer()
2020-06-05 13:09:44.359842+0100 App[12577:99329] PluginMediaStreamRenderer#init()
2020-06-05 13:09:44.360411+0100 App[12577:99329] PluginMediaStreamRenderer#run()
2020-06-05 13:09:44.360634+0100 App[12577:99329] iosrtc:MediaStreamRenderer refresh() +0ms
2020-06-05 13:09:44.360790+0100 App[12577:99329] iosrtc:MediaStreamRenderer refresh() | no video track yet +1ms
2020-06-05 13:09:44.361364+0100 App[12577:99329] [LayoutConstraints] Unable to simultaneously satisfy constraints.
    Probably at least one of the constraints in the following list is one you don't want. 
    Try this: 
        (1) look at each constraint and try to figure out which you don't expect; 
        (2) find the code that added the unwanted constraint or constraints and fix it. 
    (Note: If you're seeing NSAutoresizingMaskLayoutConstraints that you don't understand, refer to the documentation for the UIView property translatesAutoresizingMaskIntoConstraints) 
(
    "<NSAutoresizingMaskLayoutConstraint:0x6000013455e0 h=--& v=--& UIView:0x7fbeb2c77cc0.height == 0   (active)>",
    "<NSAutoresizingMaskLayoutConstraint:0x6000013393b0 h=-&- v=-&- WKWebView:0x7fbeb4818a00' App\U00ae'.minY == 20   (active, names: '|':UIView:0x7fbeb2e11c30 )>",
    "<NSAutoresizingMaskLayoutConstraint:0x600001339400 h=-&- v=-&- V:[WKWebView:0x7fbeb4818a00' App\U00ae']-(0)-|   (active, names: '|':UIView:0x7fbeb2e11c30 )>",
    "<NSLayoutConstraint:0x600001336a30 UIView:0x7fbeb2c77cc0.top == UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide'.top   (active)>",
    "<NSLayoutConstraint:0x6000013369e0 UIView:0x7fbeb2c77cc0.bottom == UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide'.bottom   (active)>",
    "<NSLayoutConstraint:0x6000013394a0 'UIView-Encapsulated-Layout-Height' UIView:0x7fbeb2e11c30.height == 667   (active)>",
    "<NSLayoutConstraint:0x600001345270 'UIViewSafeAreaLayoutGuide-bottom' V:[UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide']-(0)-|   (active, names: '|':WKWebView:0x7fbeb4818a00' App\U00ae' )>",
    "<NSLayoutConstraint:0x6000013451d0 'UIViewSafeAreaLayoutGuide-top' V:|-(0)-[UILayoutGuide:0x60000097d880'UIViewSafeAreaLayoutGuide']   (active, names: '|':WKWebView:0x7fbeb4818a00' App\U00ae' )>"
)
matijagrcic commented 4 years ago

Actually ignore the above, it works fine, using createLocalVideoTrack_ on devices running iOS12 and iOS13.

Summary:

Ionic Angular

config.xml

<plugin name="cordova-plugin-inappbrowser" spec="^3.2.0" />
<plugin name="cordova-plugin-ionic" spec="^5.4.6">
<plugin name="cordova-plugin-ionic-webview" spec="^4.1.3" />
<plugin name="cordova-plugin-wkwebviewxhrfix" spec="https://github.com/TheMattRay/cordova-plugin-wkwebviewxhrfix" />

package.json

"cordova-plugin-ionic-webview": "^5.0.0",
"cordova-plugin-iosrtc": "git+https://github.com/cordova-rtc/cordova-plugin-iosrtc.git#aee2b39b2ccebd3dc01019d7c8ebdf2bd4c90600",
"twilio-video": "^2.5.0",

I'll follow up when the Twilio P2P streaming starts working on iOS, works flawlessly on Android. Thank you for all the help and let me know if I can provide any more info that would be useful to you and the project.

matijagrcic commented 4 years ago

Both caller and participant video/audio is working on iOS in Twilio P2P room, but the video size is small. I've tried adding video constraints in terms of sizes but nothing changes the size of the video. Do you happen to know anything else I can try?

I didn't specify any height/width in the scss for the div or the video element. Is this mandatory on iOS?

On Android the video size is full height.

References: https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/332

hthetiot commented 4 years ago

I've managed to get it working on iOS 12+ (no getting ERROR: pluginMediaStream with id=13975956-A8BA-4D7D-ACB6-2FF25624A479 already exist anymore) by changing the code slightly but now it's complaining about the 2020-06-05 13:09:44.361364+0100 App[12577:99329] [LayoutConstraints] Unable to simultaneously satisfy constraints..

Layout warning can be ignored see https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/422