WebRTC Rocks(Much Android Love)

[ballenjr]

Hello,

I notice this in your readme:

Q: What about Android? Why just iOS?

R: In modern versions of Android the WebView component is based on the Chromium open source project which already includes WebRTC (more info). For older versions of Android the CrossWalk project provides new WebView versions with WebRTC support as well.

I also noticed that your constraints for getUserMedia options and the returned properties are more robust than I have seen before using Chrome. In fact some of these properties are ones that I was very hopeful would be available from the onset.

I will admit that it has been a spell since I have played with it in Chrome or Crosswalk for that matter. But my question is if Chromium is still not as elaborate/descriptive as you are natively being then would it be possible to add Android support to the plugin anyway for consistency?

I think this would be great for consistency. But of course it would probably be best as optional. So that if some one still wanted to use Chromium code then they could.

[ballenjr]

Also I am not sure if it applies but I just found out that https:// origin is being enforced now. Not sure if that precludes file:// or not. Plus it seems weird to me that wss:// wasn't mentioned too. But if it does then that would be an imperative reason to have this as an alternative to Chromium. I am assuming that it wouldn't be affected by the protocol being used on the Chromium side.

I may be testing this soon so I'll update when I know for sure. Unless someone beats me to it.

[ibc]

I also noticed that your constraints for getUserMedia options and the returned properties are more robust than I have seen before using Chrome. In fact some of these properties are ones that I was very hopeful would be available from the onset.

That is just a few lines of JS to adapt the new syntax (as per the next spec) to the old API implemented in libwebrtc.

that would be an imperative reason to have this as an alternative to Chromium.

To be absolutely clear: I won't spend my spare time adding Android support to this plugin for free. I won't even accept a Pull Request given that I won't be able to maintain such a code.

Sorry.

[ballenjr]

No worries, it probably makes more sense to have the Android implementation in a separate plugin anyway. Just thought I would see if there were any thoughts on the matter from someone that clearly knows more about the library than me.

[yocontra]

@ballenjr If you want to use WebRTC on android with cordova I would say the easiest path is to use cordova-crosswalk which gives you the latest version of chrome for your renderer, which means you have the same WebRTC chrome does. Porting this plugin to android doesn't make any sense when most android phones already have chrome by default, and for the ones who don't you can install a plugin that makes that happen.

[ballenjr]

Thanks @contra, but that's not really the point that I was going for. I acknowledged that crosswalk is available. But I am more interested in having the same implementation for both sides.

I am assuming that there is a reason that the raw datas can't be bridged into JS efficiently/accurately. Otherwise the owner would not have chosen to reimplement the video stream playing ability of the video tag and manually reimplement "some of" the methods/events/properties that go with that. Instead it would have made more sense to simply reimplement the data bearing objects and just implement the conversion/transfer across the bridge. Then on the JS side nothing would have to change.

You are correct that splitting the code to handle both sides would be easier than making the same plugin for Android.

And I would also agree that for my purpose of consistency it might not even be worth it for me to build this plugin for Android. Since I might take a different approach than the owner of this one and end up with something that doesn't quite match again.

[ibc]

Otherwise the owner would not have chosen to reimplement the video stream playing ability of the video tag and manually reimplement "some of" the methods/events/properties that go with that

As clearly explained in the doc, in order to display a WebRTC local/remote video associated with a MediaStream, the plugin places a native iOS UIView on top of the web view, so the HTML <video> element does not handle a real Web video stream, and that's why obviously the video properties/methods are overridden by the plugin's JavaScript code.

[ballenjr]

Yup

[ballenjr]

Also so you know. One way that I have seen others get around putting it on top is by putting it under and making just the area that they want to not be covered transparent. Like the mapsplugin now being curated by @hirbod.

This allows any opaque html elements to be rendered on top of the surface of the map or in your case video.

Not sure how/if he did it succesfuly on iOS though. And I imagine that there may be more of a challenge for the video tag since there is the potential for multiple tags in a scrollable. Like I am using in another project. Where as the map typically only has to worry about one surface that may typically span the whole screen making it easier to only cover what you want to. On Dec 20, 2015 5:09 PM, "Iñaki Baz Castillo" notifications@github.com wrote:

Otherwise the owner would not have chosen to reimplement the video stream playing ability of the video tag and manually reimplement "some of" the methods/events/properties that go with that

As clearly explained in the doc, in order to display a WebRTC local/remote video associated with a MediaStream, the plugin places a native iOS UIView on top of the web view, so the HTML

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-166170070 .

[ballenjr]

But then again for the purpose of video chat it may be possible to just combine each feed onto one surface. And there would probably be no need to ever scroll to any video tags.

[ballenjr]

Doesn't iOS have something akin to the SurfaceView in AVFoundation?

[1N50MN14]

You guys are missing the main point here. For anyone to be able to successfully use this plugin in production you need to rely on H264 otherwise you'll get a laggy video experience using VP8 due to the lack of HW acceleration. All good and fine as @ibc just added support for that otherwise you can build libwebrtc yourself like I did.

Problem is, If you use H264 on iOS, you also need WebRTC on Android to rely on H264 to establish a successful video session between the two, and that is not supported in Chromium yet and I don't expect it to go into Crosswalk anytime time soon. Realistically speaking I'd say 12 months and up over, by then libwebrtc is already far ahead and we'd end up playing catch-up again.

So having and "Android" version of this plugin is a must for serious production use. @ibc has plenty on his plate already and won't be able to maintain such a project. As such, I have started porting this plugin to Android, the workis 99% complete and it includes an Android libwebrtc build with H264 support. This can either published as a PR to this plugin or as a separate plugin, the main obstacle here as I see it is whether we have enough people who are interested and are willing to contribute to such a thing.

@ibc let me know what you think and I'll act on that, your opinion is important because I practically re-used all of your .js files (java bridges files obviously written from scratch).

[ibc]

@1N50MN14 please let me see your code (is it in Github or somewhere?). I cannot promise nothing for now since, as already say, I don't know whether I will be able to maintain Java code, but we can talk about that.

[1N50MN14]

@ibc It's still in a private repo got some fixes to do (placing the views behind the webview, correct dimensions of video views etc), will get that out of the way, do some clean up and push it to my fork instead then you can decide whether you want to merge it or not. This will be 100% your call.

[ibc]

Great.

[ballenjr]

That's cool that the codecs can be available on both sides too. But I was under the impression that one of the two didn't have hw acceleration for h264? Or am I dating my self? I was thinking it was Android.

Honestly that was least if my concern anyway cuz I thought that the stack negotiated that on its own. Or does it actually default to VP8? On Dec 22, 2015 7:09 AM, "Ayman Mackouly" notifications@github.com wrote:

You guys are missing the main point here. For anyone to be able to successfully use this plugin in production you need to rely on H264 otherwise you'll get a laggy video experience using VP8 due to the lack of HW acceleration. All good and fine as @ibc https://github.com/ibc just added support for that otherwise you can build libwebrtc yourself like I did.

Problem is, If you use H264 on iOS, you also need WebRTC on Android to rely on H264 to establish a successful video session between the two, and that is not supported in Chromium yet and I don't expect it to go into Crosswalk anytime time soon. Realistically speaking I'd say 12 months and up over, by then libwebrtc is already far ahead and we'd end up playing catch-me again.

So having and "Android" version of this plugin is a must for serious production use. @ibc https://github.com/ibc has plenty on his plate already and won't be able to maintain such a project. As such, I have started porting this plugin to Android, the workis 99% complete and it includes an Android libwebrtc build with H264 support. This can either published as a PR to this plugin or as a separate plugin, the main obstacle here as I see it is whether we have enough people who are interested and are willing to contribute to such a thing.

@ibc https://github.com/ibc let me know what you think and I'll act on that, your opinion is important because practically re-used all of your .js files (java bridges files obviously written from scratch).

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-166641051 .

[ballenjr]

I'm no master at the protocol yet. But I'm pretty good with Android and Java code. And I did spend a couple months getting as familiar as I could with the one and outs of webrtc at the surface. So I could help out if needed. I wouldn't be offended if you declined though. On Dec 22, 2015 7:09 AM, "Ayman Mackouly" notifications@github.com wrote:

You guys are missing the main point here. For anyone to be able to successfully use this plugin in production you need to rely on H264 otherwise you'll get a laggy video experience using VP8 due to the lack of HW acceleration. All good and fine as @ibc https://github.com/ibc just added support for that otherwise you can build libwebrtc yourself like I did.

Problem is, If you use H264 on iOS, you also need WebRTC on Android to rely on H264 to establish a successful video session between the two, and that is not supported in Chromium yet and I don't expect it to go into Crosswalk anytime time soon. Realistically speaking I'd say 12 months and up over, by then libwebrtc is already far ahead and we'd end up playing catch-me again.

So having and "Android" version of this plugin is a must for serious production use. @ibc https://github.com/ibc has plenty on his plate already and won't be able to maintain such a project. As such, I have started porting this plugin to Android, the workis 99% complete and it includes an Android libwebrtc build with H264 support. This can either published as a PR to this plugin or as a separate plugin, the main obstacle here as I see it is whether we have enough people who are interested and are willing to contribute to such a thing.

@ibc https://github.com/ibc let me know what you think and I'll act on that, your opinion is important because practically re-used all of your .js files (java bridges files obviously written from scratch).

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-166641051 .

[1N50MN14]

@ballenjr H264 is pretty standard on Android, yes you will need to mangle your SDP template and let it know which codec to prioritise. YES, help will be welcome I will surely ping you! :-)

[ballenjr]

See, that's what gets me. If it's accelerated and there is already priority ranking available why wouldn't it be prioritized. And why aren't there already helper methods for doing the useful mangles like weighting codecs. It's too easy to get that packet messed up as it is. I hope at least the handshake is more streamlined now. On Dec 22, 2015 7:43 AM, "Ayman Mackouly" notifications@github.com wrote:

@ballenjr https://github.com/ballenjr H264 is pretty standard on Android, yes you will need to mangle your SDP template and let it know which codec to prioritise. YES, help will be welcome I will surely ping you! :-)

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-166650834 .

[1N50MN14]

@ballenjr Agree with you, as someone who's relatively new to webrtc those were exactly my thoughts about the lack of such helper methods, more precisely, lame and lazy specially considering how "simple" WebRTC was promoted to world.

[ballenjr]

No doubt. On Dec 22, 2015 8:47 AM, "Ayman Mackouly" notifications@github.com wrote:

@ballenjr https://github.com/ballenjr Agree with you, as someone who's relatively new to webrtc those were exactly my thoughts about the lack of such helper methods, more precisely, lame and lazy specially considering how "simple" WebRTC was promoted to world.

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-166670640 .

[ibc]

And why aren't there already helper methods for doing the useful mangles like weighting codecs.

Say thanks to telcos for forcing SDP into WebRTC. SDP is a unmanageable "blob" string. And, even if you can mangle it, you cannot be sure whether the browser will accept those changes between getLocalDescription() and setLocalDescription().

ORTC will change that when it becomes WebRTC 2.0.

[ballenjr]

Greedy bastards.

[1N50MN14]

Hey guys, I think I need some help from couple Java/Android gurus to give the Android port a final push the goal is to completely nail and finish testing the port within 7-10 days. The challenges are now mostly down to Java/Android skills / experience which means it's time for better developers to step in.

To make this process efficient I've set up a Slack channel (with two members right now), I'm looking for couple guys so that we're 3 + me in total which is just about ideal to sync and move efficiently on this task.

@ballenjr Are you still up for this? @contra You're welcome to join too @ibc I know you're busy, you're spared :-)

[ballenjr]

I am pretty busy with something right now. But I will very likely be needing this in the future. So I do want to help.

I take it you don't want discuss what you need help with here. So I am just wondering what the plans are for opening up the source?

[ballenjr]

@1N50MN14, did you already find the help that you needed? Or did this get set aside for now?

[1N50MN14]

@ballenjr I put it aside for now, will get back to it in few weeks

[ballenjr]

Cool, let me know. I'd love to help. On Jan 2, 2016 2:46 PM, "Ayman Mackouly" notifications@github.com wrote:

@ballenjr https://github.com/ballenjr I put it aside for now, will get back to it in few weeks

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-168435958 .

[ballenjr]

What if the client didn't want to pay for h264? Does libwebrtc have any other codecs in common with android/chromium? I couldn't find any docs.

[ibc]

Does libwebrtc have any other codecs in common with android/chromium? I couldn't find any docs.

libwebrtc is the WebRTC library Google uses for Chrome/Chromium and related stuff.

[ballenjr]

I apologize for the mis phrasing of the question. It's my understanding that the codecs are pluggable. Are you saying that the all of the supported codecs are included with libwebrtc? On Feb 15, 2016 1:43 AM, "Iñaki Baz Castillo" notifications@github.com wrote:

Does libwebrtc have any other codecs in common with android/chromium? I couldn't find any docs.

libwebrtc is the WebRTC library Google uses for Chrome/Chromium and related stuff.

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-184135228 .

[ibc]

2016-02-16 1:52 GMT+01:00 ballenjr notifications@github.com:

I apologize for the mis phrasing of the question. It's my understanding that the codecs are pluggable. Are you saying that the all of the supported codecs are included with libwebrtc?

Yes

Iñaki Baz Castillo ibc@aliax.net

[1N50MN14]

No that's a misleading answer. @ballenjr You can follow on this thread https://code.google.com/p/chromium/issues/detail?id=500605, read it carefully specially mid way, it'll provide you with the information you're looking for, depending on what codecs you decide to use and how you use them you may end up needing to pay (rather expensive) royalty fees to MPEGLA, as such you need to be careful and try to work around those issues (for example such as getting your application to download OpenH264 as a third party binary from Cisco). On iOS you don't have this issue as libwebrtc relies on the codec available in iOS AVTools (or whatever it's called) and there Apple has already covered the royalty fees of MPEGLA.

[ibc]

Thanks for the fix. What I really wanted to mean is that, AFAIK, there is no way to set specific codecs with the ObjC wrapper of libwebrtc.

[ballenjr]

Funny, that's the same thread that prompted me to ask about which codecs were included on both...

[1N50MN14]

@ibc True, for the plugin as it is there's nothing that needs to be done, H264 is supported on HW level on iOS and libwebrtc utilizes that natively. @ballenjr If you're going to rely on H264 for both iOS and Android to talk to each other, you have three options: 1) Gamble on native HW H264 support which may or may not be dependable based on the underlying HW and whether it supports H264 at all without a software codec 2) Get your android app to download openH264 as a third party binary and use that 3) Pay MPEGLA licensing fee

[1N50MN14]

@ballenjr libwebrtc does support both VP8 and H264 (OpenH264) which you can either compile from source or use as a binary. Those are the supported codecs as of today, VP9 is being worked on it.

[ballenjr]

Ya, what I got from the thread to was that the fee would be covered if the system webview is used instead of chromium wrapped with crosswalk.

But how could apple have the fees covered when using libwebrtc as a library? That same logic from the thread is what had me thinking that this would not be the case and that fees would apply.

[ballenjr]

That's what I was hoping. I was just thinking that it was only chrome that supported vp8 independently as opposed to it being included with libwebrtc. On Feb 16, 2016 1:30 AM, "Ayman Mackouly" notifications@github.com wrote:

@ballenjr https://github.com/ballenjr libwebrtc does support both VP8 and H264 (OpenH264) which you can either download from source or as a binary. Those are the supported codecs as of today, VP9 is being worked on it.

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-184596315 .

[ibc]

But how could apple have the fees covered when using libwebrtc as a library?

You create an iOS app that may include libwebrtc. Since iOS allows its app to use built-in codecs the app (and hence its libs) can make use of native h264.

[1N50MN14]

@ballenjr True, problem is there as of today there are 0% system web views on Android that will give you that, libwebrtc with OpenH264 support hasn't landed in Chrome yet. On iOS, libwebrtc relies on native H264 through Apple's API's that already exposed for developers.

[ballenjr]

Ahh... Now it makes sense. Cool! On Feb 16, 2016 1:35 AM, "Iñaki Baz Castillo" notifications@github.com wrote:

But how could apple have the fees covered when using libwebrtc as a library?

You create an iOS app that may include libwebrtc. Since iOS allows its app to use built-in codecs the app (and hence its libs) can make use of native h264.

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-184597647 .

[ballenjr]

0? You mean h264, right? That's true. But I was thinking if fees were going to apply anyway then I would just go with vp8. It worked pretty good in my initial tests. But if it's free on apple too... I might just roll with vp8 then when m50-ish comes out I'll prioritize h264 with vp8 fallback when hw support is dicey. On Feb 16, 2016 1:36 AM, "Ayman Mackouly" notifications@github.com wrote:

@ballenjr https://github.com/ballenjr True, problem is there as of today there are 0% system web views on Android that will give you that, libwebrtc with OpenH264 support hasn't landed in Chrome yet. On iOS, libwebrtc relies on native H264 through Apple's API's that already exposed for developers.

— Reply to this email directly or view it on GitHub https://github.com/eface2face/cordova-plugin-iosrtc/issues/109#issuecomment-184597781 .

[ballenjr]

Assuming there is a reliable way to accomplish that.

[1N50MN14]

@ballenjr Not so fast :) VP8 isn't supported on HW level on iOS you're probably going to end up with a laggy experience, then there's the risk of rejection from Apple I'm not sure they're keen on anything that sounds "VP8". Submitting an app that relies on webrtc might be tricky to start with, but they have been slightly more flexible there. Also if you intend to transcode/record your videos, you're better off with H264 then you can playback mp4 files on both Android and iOS, that's more widely supported. To avoid paying licensing fees your best bet (in my humble opinion) is to do what I do: native H264 for iOS + OpenH264 as third party binary or rely on HW for Android

[ballenjr]

Ya good point. For some reason I was thinking that vp8 wasn't baked in hw on Android either. And that the experience would be consistent.

[ballenjr]

Probably still isn't for some older kitkat thought. You think? On Feb 16, 2016 1:53 AM, "Brent Allen" brent.allen.ba@gmail.com wrote:

Ya good point. For some reason I was thinking that vp8 wasn't baked in hw on Android either. And that the experience would be consistent.

[1N50MN14]

@ballenjr If you use Crosswalk you'll have VP8 support on Android 4 and upover

[ballenjr]

Cool, thanks for the clarification and the tips.

[ramsestom]

@1N50MN14 Is your version supporting android public now? I am trying to make a cordova android/ios app that uses webrtc but I didn't succeed to make it work on android (even with crosswalk). It appear that the camera is not found on my android device (DevicesNotFoundError) for some strange reason (I am probably missing something here but I don't know what). So I was hopping that a unified android/ios plugin would fix the issue ;) Anyway, if someone has a working example of usage of webrtc on android with cordova (with crosswalk), I will take it.