Video Sometimes Not Shown when using Apizee

[AleksandarTokarev]

Hello, We are seeing issues with the Video on various IPhone devices running on different IOS Versions (12-14 at the time) When app is initially opened (from killed state) and a video call is established, the IOS video is OK and running. After repeating the call with the app open several times, video starts to not show. There are several states that i am seeing:

1. Video sometimes does not show only on the IOS device (white screen)- but it shows for the other person on the call
2. Video sometimes does not show for both sides (white screen instead of video)
3. Video sometimes is black for both sides

In all of the cases, Audio is working fine.

We are using Apizee as WebRTC provider (https://www.apizee.com/)

Also in the logs sometimes we are seeing this error

Error: getStats() must be called with null or a valid MediaStreamTrack instance as argument", url: "http://localhost:12710/plugins/cordova-plugin-iosrtc/dist/cordova-plugin-iosrtc.js", lineNo: 2263, columnNo: 106, error: {}, stack: "http://localhost:12710/plugins/cordova-plugin-iosrtc/dist/cordova-plugin-iosrtc.js:2263:18\nhttps://cloud.apizee.com/apiRTC/apiRTC-latest.min.js:2:651202"

Versions affected

• Plugin version (6.0.11, 6.0.14, 8.0.0-RC2):
• iOS version (13.X):
• Xcode version (11.6):

Steps to reproduce

Start a Video Call several times in a row

Expected results

Video should always show up

Actual results

Video sometimes does not show up

[hthetiot]

Sorry this is most likely related to Apizee usage of WebRTC itself that is not in the officially supported list of libraries. We are investigating MediaStream recyclying issue that may be related to what you experience but we also found it itmostly depend the way you use WebRTC and manage MediaStream (stopping them) and VideoMediaElement (empty them).

1. " I have provided steps to reproduce (e.g. sample code or updated extra/renderer-and-libwebrtc-tests.js file)." https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/.github/ISSUE_TEMPLATE.md

You did not provide step to reproduce your issue using Apizee API.

2. "If the checkboxes above are not checked (which you do after the issue is posted), the issue will be closed, removing this checkbox will result in automatic closed issue."

You removed checkboxes and therefor the issue is closed, as you would have seen if you had read the ISSUE requirement.

You can always update this issue to match requirement (mainly "I have provided steps to reproduce"), but it unlikely I will help since i not paying or using Apizee and it will be to Apizee itself or the community to assist you with a fix once you updated the issue and its re-openned.

[hthetiot]

Error: getStats() must be called with null or a valid MediaStreamTrack instance as argument", url: "http://localhost:12710/plugins/cordova-plugin-iosrtc/dist/cordova-plugin-iosrtc.js", lineNo: 2263, columnNo: 106, error: {}, stack: "http://localhost:12710/plugins/cordova-plugin-iosrtc/dist/cordova-plugin-iosrtc.js:2263:18\nhttps://cloud.apizee.com/apiRTC/apiRTC-latest.min.js:2:651202"

Visibly they dont know how GetStats work since it does take a MediaStream or null... since 2014 (6 years ago):

• https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection/getStats

[hthetiot]

Never heard about white screen before in my years of webrtc, i guess that Apizee MCU/SFU doing that...

[hthetiot]

It will be inestigated once https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/575 that is most likely a duplicate

[AleksandarTokarev]

@hthetiot Apizee = ApiRTC. This is stated in the "Who Uses It" section https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/WHO_USES_IT.md I guess it should be supported by this plugin?

One thing we noticed is, this thing always happens on the Callee side. If we have an user with IPHONE calling an user with IPHONE, the video and the stream ALWAYS works on the Caller side (means caller gets his own video and the Caller video is displayed on the Callee). On the callee side, the first 2-3 times it works but after that the video does not (means the Calle e video is not shown on Callee side and Callee video is NOT shown on Caller side as well). We are basically using the 1 on 1 call scenario specific here https://dev.apirtc.com/tutorials/peertopeer/call

 .on("localStreamAvailable", function (stream) {
            console.log('localStreamAvailable');
            //document.getElementById('local-media').remove();
            addStreamInDiv(stream, 'local-container', 'local-media-' + stream.getId(), {width : "160px", height : "120px"}, true);
            stream
                .on("stopped", function () { //When client receives an screenSharing call from another user
                    console.error("Stream stopped");
                    $('#local-media-' + stream.getId()).remove();
                });
        })
        .on("streamAdded", function (stream) {
            console.log('stream :', stream);
            addStreamInDiv(stream, 'remote-container', 'remote-media-' + stream.getId(), {width : "640px", height : "480px"}, false);
        })
        .on('streamRemoved', function (stream) {
            // Remove media element
            document.getElementById('remote-media-' + stream.getId()).remove();
        })
        .on('userMediaError', function (e) {
            console.log('userMediaError detected : ', e);
            console.log('userMediaError detected with error : ', e.error);

            //Checking if tryAudioCallActivated
            if (e.tryAudioCallActivated === false) {
                $('#hangup-' + call.getId()).remove();
            }
        })
        .on('desktopCapture', function (e) {
            console.log('desktopCapture event : ', e);
            $('#hangup-' + call.getId()).remove();
        })
        .on('hangup', function () {
            $('#hangup-' + call.getId()).remove();
        });

Also what you posted above, i can confirm that the MediaStreamObjects are piling up if subsequent calls are being made. I tried to do cordova.plugins.iosrtc.mediaStreams = undefined but nothing changed, the behavior is same

[hthetiot]

@hthetiot Apizee = ApiRTC. This is stated in the "Who Uses It" section https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/WHO_USES_IT.md I guess it should be supported by this plugin

In fact it does not implicitly make them supported, it simply that they use it.

Thank you for providing way to reproduce, let see if 6.0.14 fix your problem, if not we can try to address it in 6.0.15. please try master this way to confirm if incoming 6.0.14 fix some of your problems.

[AleksandarTokarev]

I tried the latest master commit. This seems even more unstable for me now. Before the caller side was working OK, now with 6.0.14, that seems unstable as well.

[hthetiot]

arf thank you @AleksandarTokarev can you elaborate by what you mean by "even more unstable", this is not clear for me. master is stable for me.

[hthetiot]

Here is 6.0.13 vs master changes for references: 6.0.13.diff.txt

+ Move dist/cordova-plugin-iosrtc.js to www/cordova-plugin-iosrtc.js to match other cordova plugins practice. + fix: undefined references #563 + fix: use video view for correct screencap framing #570 + fix: cleanup old streams #570 + implement basic MediaDevices.prototype.getSupportedConstraints SHIM #564 + fix RTCPeerConnection.prototype.addTrack missing return RTCRtpSender #572

[hthetiot]

I think your issue is only when you make multiple calls, and only relate to https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/575 that will be treated in 6.0.15

[AleksandarTokarev]

With the newest master version (which is supposed to be 6.0.14), the behavior is different as you can see in the video below. Basically, the Remote Stream is not available at all - on both devices. After some calls, on the caller side, the Local Stream is not available as well https://www.dropbox.com/s/ap7ibahzuacbbke/20201017_182151.mp4?dl=0

And yes, on 6.0.11, this happens when multiple calls are being made.

[hthetiot]

Thank you @AleksandarTokarev that more clear to me, I will do what I can to fix this issue on 6.0.15

[AleksandarTokarev]

@hthetiot Here is a video of 6.0.11 behavior. Quite different than 6.0.14. Here the Remote video is available first 2-3 times, but from 3-4th on, the Callee stream is not available at all https://www.dropbox.com/s/qr0lx2vfxtevn12/20201017_184053.mp4?dl=0

[hthetiot]

@AleksandarTokarev can you try this PR https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/587 This , I think should change the behavior to a better result. Keep me posted.

[AleksandarTokarev]

I tried #587 @hthetiot , it loads the video on both sides all the time now. Here is a video of how it works now. https://www.dropbox.com/s/unc8e8v86u7gra5/20201018_100612.mp4?dl=0 One small comment which is not so important - but good if it can be addressed for better UI/UX - would be the black screen not to show initially (if it is possible - check the video in the comment).

Thanks a lot @hthetiot, i really appreciate your help!!!!

[hthetiot]

@AleksandarTokarev I think this PR will address the black screen background you see in your video. Will be merged soon.

https://github.com/cordova-rtc/cordova-plugin-iosrtc/pull/586

[hthetiot]

Thank you for taking videos that was really useful to understand your issue, also nice UI.

Happy RTC.

[AleksandarTokarev]

@hthetiot Is there a way to test #586 and #587 together before being merged into master? Making a custom branch with both things would be possible, but not sure if the repo is managed like that. Thanks a lot for all your help!!!!

[hthetiot]

I can make a branch for you, give me a moment. Will notify you here.

[hthetiot]

@AleksandarTokarev Here is your branch (https://github.com/cordova-rtc/cordova-plugin-iosrtc/tree/AleksandarTokarev) with #586 and #587 together, notice that i have not tested #586 yet, you may try to set the background color of the video tags via CSS in your pages like this video {background-color: purple; } to test it.

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#AleksandarTokarev --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

How i did it, in case you want to do custom branch in the future:

git clone git@github.com:cordova-rtc/cordova-plugin-iosrtc.git
cd cordova-plugin-iosrtc
git remote add calebboyd https://github.com/calebboyd/cordova-plugin-iosrtc.git
git fetch calebboyd -a -v
git checkout -b AleksandarTokarev
git merge bugs/PeerConnectionTracks
git merge background
git push --set-upstream origin AleksandarTokarev

Thank you, your testing of #586 will save me some time ;)

[hthetiot]

Feel free to add your company/app to https://github.com/cordova-rtc/cordova-plugin-iosrtc/blob/master/WHO_USES_IT.md when you want :)

[AleksandarTokarev]

@hthetiot I tried the branch you created for #586 and #587 , it seems that the background-color stuff works fine, but the remote video is not available (as it was in the 6.0.14). Here is a video of that https://www.dropbox.com/s/hzybgso1qmc51nz/20201018_121745.mp4?dl=0

Also as u can see, sometimes a black screen is shown initially. I only added background-color: purple; in the code

[hthetiot]

Thank you @AleksandarTokarev, the version for you will actually be 6.0.15 aka master since I just landed #586 on master and released 6.0.14.

@calebboyd can you check the branch #586 and rebase/merge on master, I suspect the issue is your "superView" extras view that break HTML over video.

[hthetiot]

If you dont mind @AleksandarTokarev try master one most time to make sure we get same result as when you tested #587 Thank you, your testing help a lot.

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#master --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

[AleksandarTokarev]

@hthetiot i tested the latest master (without the last commit u had made just now) and apart from the black screen initially (which i reported in https://github.com/cordova-rtc/cordova-plugin-iosrtc/issues/583#issuecomment-711133150) , all looks good. Here is the commit that i tested with

[hthetiot]

Perfect, yes those commit are for README and sample and does not have impact for you. I just fixed the jslint issue on render-and-libwebrtc-test.js ;)

[AleksandarTokarev]

Thanks a lot for everything. If it is possible to fix the black screen in the future it would be great. Thanks a lot and let me know if i need to do some testing in the future :)

[AleksandarTokarev]

@hthetiot We did some more thorough testing with our colleagues today and it seems that the problem still appears (But not as often). Before the changes - after like 3rd time, the Callee video was not available in any sub sequent calls. Now this is not the case, which basically means maybe after 5-6 times, 7th maybe it wont appear (the Callee video), but after that it starts appearing again.

[hthetiot]

Unfortunaly i do not reproduce using the iosrtc sample, what would really help to confirm this is an iosrtc issue would to create an index-apizee.js file in the sample to test the iosrtc sample with apizee.

See existing samples:

• https://github.com/cordova-rtc/cordova-plugin-iosrtc-sample/tree/master/www/js

Using JsSip and Local and EasyRTC i can make 20 calls without issues.

[hthetiot]

Regarding yout black screen; you can also handle it by displaying a loader until you receive video event "canplay".