OpenTok / TokBox support

[noodlemichaelmartin]

After compiling and running it on my iPhone, I attempted to load a page that I've been working on that uses the OpenTok/TokBox libraries (WebRTC-based video chat platform).

The page loads with no errors, but the pre-detection of whether or not WebRTC is present fails. The variables/objects that it's looking for are below:

if (!navigator.mozGetUserMedia && !navigator.webkitGetUserMedia) {
...
return false;
}

If I remove the pre-detection, I get a JS error that prevents the page from fully loading.

I'm assuming that perhaps the functions are implemented differently in Bowser, so I have the following questions about the WebRTC implementation in Bowser:

1) Were navigator.mozGetUserMedia or nagivator.webkitGetUserMedia implemented under different object/method names?

2) Is there any documentation regarding the potential differences in building web pages for Bowser?

3) Where in the source would the definition of the methods attached to the nagivator object be found?

It's possible that TokBox/OpenTok would not be compatible with your WebRTC implementation or that perhaps only some minor shims would be required to use the correct object/method names - I'm just trying to determine in which direction I should go.

[stefanalund]

@noodlemichaelmartin "Bowser/0.2.1" is included in the UserAgent so you could just check for that. I leave the other question to someone else (ping @adam-be @pererikb).

[webvinita]

Dear Noodlemichaelmartin

Are you able to make opentok working on iPhone Bowser. I have been also waiting for a browser which can be used in iPhone to run my opentok application. Any suggestion on how to make openok application working on iPhone Bowser?

[EvilTrev]

Did anyone make any further progress on this? I've been stabbing at trying to solve the same issue, so far:

1. I've had to include http://localhost:10717/owr.js myself in the head area of my page, because otherwise other loading code cannot find getUserMedia etc... it's all undefined.

2. Include this code immediately after the above header to patch in the missing prototype. It would seem Bowser acts like webkit, but not like Chrome/Opera... I've seen others using this same prototype for testing the presence of webrtc... so not just OpenTok. I'm sure this is old/invalid or not to standard... but it exists in the most recent version of OpenTok and Chrome/Opera seems to allow and pass on the test for it.

   if(window.webkitRTCPeerConnection) {
      var tempHack = new webkitRTCPeerConnection({iceServers:[]});
      window.webkitRTCPeerConnection.prototype.addStream = tempHack.addStream;
   }

3. I replaced self.javascriptCode with an empty @"" string in BowserViewController.m to avoid causing issues with including it manually.

4. For an experiment, I also hacked opentok.js to include the code below when they create the peerconnection (this was just in case there is some magic lost in the above stub... it didn't seem to make any difference):

   pc = new NativeRTCPeerConnection(config, options);
   if (window.webkitRTCPeerConnection) window.webkitRTCPeerConnection.prototype.addStream = pc.addStream;

5. Also tried playing with the useragent string (which wasn't required, but thought it might prevent some monkeying)... no change.

   #define kUserAgent @"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36"
   
   #define kUserAgent @"Mozilla/5.0 (iOS; like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) not Chrome/36.0.1500.95 Mobile/12B411 Safari/600.1.4 Bowser/%@"

6. Found this code explaining the setEnabled nonsense (basically it's to support OpenTok IE plugin):

   // The setEnabled method on MediaStreamTracks is a OTPlugin
   // construct. In this particular instance it's easier to bring
   // all the good browsers down to IE's level than bootstrap it up.
   if (typeof window.MediaStreamTrack !== 'undefined') {
      if (!window.MediaStreamTrack.prototype.setEnabled) {
          window.MediaStreamTrack.prototype.setEnabled = function (enabled) {
              this.enabled = OT.$.castToBoolean(enabled);
          };
      }
   }

7. Wrapped the opentok.js setEnabled in try...catch, found that got local video preview to show, remote doesn't, seems to be stopped dead by these exceptions:

   [Error] Failed to load resource: unsupported URL (mediastream:tOtTiip/yEO0ZWmtk9XMhrfxXeWujnElFMf2, line 0)
   [Error] OT.exception :: title: Connection Failed (1013) msg: Publisher PeerConnection Error: There was an unexpected problem with the Video Stream: The media resource indicated by the src attribute was not suitable.
   [Error] 1013 "Publisher PeerConnection Error: There was an unexpected problem with the Video Stream: The media resource indicated by the src attribute was not suitable."
   [Error] OT.exception :: title: Connection Failed (1013) msg: Publisher PeerConnection Error: There was an unexpected problem with the Video Stream: The media resource indicated by the src attribute was not suitable.
   [Error] 1500 "Session.publish :: Publisher PeerConnection Error: There was an unexpected problem with the Video Stream: The media resource indicated by the src attribute was not suitable."
   [Error] OT.exception :: title: Unable to Publish (1500) msg: Session.publish :: Publisher PeerConnection Error: There was an unexpected problem with the Video Stream: The media resource indicated by the src attribute was not suitable.

8. Found possible clue to the above spew (haven't had chance to investigate):

   In Chrome or Opera, the URL.createObjectURL() method converts a MediaStream to a Blob URL which can be set as the src of a video element. (In Firefox and Opera, the src of the video can be set from the stream itself.)

9. Plenty of this going on in Xcode:

   (<unknown>:12691): GStreamer-CRITICAL **: guint64 _gst_util_uint64_scale(guint64, guint64, guint64, guint64): assertion `denom != 0' failed

All pretty hacktastic so far... I'm just trying to prove that it might even work. Which it doesn't currently :) I don't see the iPhone on my desktop, I don't see the desktop on the iPhone, the preview on the iPhone is super laggy, the iPhone seems to hear itself (not feedback screams, just echos). When the session is terminated, the phone still hears itself. I'll report back if I make any further progress.

[EvilTrev]

So no? ;)

[superdump]

@adam-be?

@EvilTrev - we will get to testing more thoroughly such WebRTC apps and figuring out problems but our priority so far has been on improving the core of the framework and project infrastructure (new build system coming very soon, automated builds, releases, testing).

We plan to compile a list of popular WebRTC libraries commonly used by apps/services, test compatibility with those and fix/report issues.

[RyanPetrucci]

Hey!

Any updates on this?

As far as I can currently see this is by far the nearest solution to OpenTok support within a hybrid iOS app - what's the outlook on adding OpenTok support any time soon?

@noodlemichaelmartin - as you never returned does that mean you found a solution or also are currently awaiting an update on this?

Cheers!! :)

[noodlemichaelmartin]

We have not found a solution. Another developer here attempted to build a more recent version of Bowser last month and was met with device support issues that render the browser unusable for us and the majority of our potential users.

We've been forced to abandon this issue until such time when either Bowser runs on a more expansive list of "legacy" iOS devices or appropriate solution presents itself.

[RyanPetrucci]

Hey!

Thanks a lot for the update, I've been searching and searching for a hybrid iOS WebRTC solution and was sure I'd found one until now.

It's a shame Apple force vendors to use only their rendering engine else we'd have a solution with Chromium.

On that front I wonder if it would be viable to create a fork of Chromium and add WebRTC support using Google's WebRTC library in a similar to manner that EricssonResearch added WebRTC support using the OpenWebRTC library or if it's already in the works - if so it's hiding well!

If I do manage to find/create a solution I'll let you know and leave an update on here anyway as I know this is widely sought.

Thanks for the fast response! :)

[superdump]

The device support issue is a problem with language bindings and the changing hardware landscape - arm64 entered the picture, libffi was updated to support arm64 on iOS. This simultaneously broke armv7 support on iOS and it is yet to be fixed. Unfortunately as the iOS toolchain has also been updated by Apple, it is not a case of just using an old version of libffi for armv7 iOS until it is fixed as they no longer work either with newer Xcode.

[RyanPetrucci]

Basically we got an inch away from WebRTC on native iOS apps and Apple killed it - good to know!

If a solution crops up in the medium-terms I'll report back here unless somebody else does - thanks for the clarification and quick reply! :)

[noodlemichaelmartin]

@superdump - Yes, that's what he ran into when trying to get Bowser to run on several of our test devices.

@ArtisanCodesmith - Yes, we were excessively close during my initial attempts. Got down to some JS issues that we never quite got resolved. However, good luck to you on the Chromium+OpenWebRTC idea; hope you manage to get it working. We opted not to pursue a custom solution, as we would need to develop and support the app ourselves. TokBox has an iOS SDK, but it's not a browser and is rather myopic - you can't easily have web content + video chat in one place without, essentially, building your own browser within your custom app. Using a normal browser on iOS, we'd basically have needed to embed links to the SDK-based external, custom app in the web page and force the user to switch between apps to make the whole thing work. That's a terrible user experience. Only a browser with some form of webRTC technology embedded into it provides the appropriate user experience.

[RyanPetrucci]

They're currently requesting audio and data support in Chromium for iOS here which the Chromium team has said would be achieved easily which would be great as it would be half-way there at least. Will have to wait on that (and perhaps nag a little as then there would be something to perhaps work with).

They apparently will then opensource the web shell which will open a few doors also.

I'll keep looking into solutions and report back anyway - thanks and good luck with everything! :)

[superdump]

We're actively making patches for a project to add WebRTC support into WebKit and some have landed already. http://www.webrtcinwebkit.org/

Maybe in the not too distant future Apple will decide to jump onto the mobile WebRTC train.

[RyanPetrucci]

That's great and most definitely the way to go - having WebRTC support built into WebKit would be outstanding! Thanks for actively contributing to such a feat!

[RyanPetrucci]

Hey!

I've successfully made an iPad app using OpenTok that allows for group video chats (the videos can be dragged around and all sorts!). You have to update the videos' position when dragged manually but it's incredibly smooth!

It has the same codebase for Android and the web also (the Android version is wrapped in CrossWalk which has full WebRTC support).

Hopefully you can get your app finished now @noodlemichaelmartin !

To achieve this I used PhoneGap and this Cordova plugin!

The video/voice chat now works seamlessly across iOS, Android and browsers!

This does not work with PhoneGap Build I must add and you must also (in order to build the app) first prepare it via the terminal by doing: "phonegap prepare ios"

Then open the created Xcode project in Xcode and finish the build yourself and tadaa!

A fully working WebRTC app on the iOS!

Really hope this helps you get back to the project you're working on - it you need any help give us a shout and I'll be more than happy to oblige!

See you soon!! :)

@webvinita @EvilTrev - you two also mentioned you have projects that require fully functional OpenTok on the iOS - I hope this helps!