Open noodlemichaelmartin opened 10 years ago
1) navigator.getUserMedia
should be present and it seems we also overwrite navigator.webkitGetUserMedia
: https://github.com/EricssonResearch/openwebrtc/blob/master/bridge/client/webrtc.js#L1224 (note there and just below, also note the FIXME)
2) Not yet but that is a good idea. I think we may be able to put that together and it would help in general.
3) See above. bridge/client/webrtc.js it seems.
Thanks for the feedback, we're definitely interested in either fixing bugs to allow interoperability with higher-level WebRTC application frameworks or to investigate them to get them to fix mistakes in WebRTC API usage.
If you are comfortable, and can, share a URL for your app we can also take it for a spin.
Unfortunately, I cannot - it's not public.
I can say that we're using the standard implementation of TokBox, which is basically a paid version of the "OpenTok" standards (they host the TURN/STUN services and all the streams are routed through their Media server to enable things like recording and changes in frame rates or stream resolution).
If you want to test the libraries themselves, they do let you setup a free 30-day trial developer account prior to setting up billing - https://tokbox.com/
The only thing that I do in my implementation that's not present in the out of the box TokBox implementation, is the above JS detection to put a custom message into the DOM for the user. When I remove the JS detection code, it causes a JS error that prevents the rest of the JS on the page from executing. I can't seem to see what the error is, though. It should be noted that while the navigator object is defined, neither the navigator.mozGetUserMedia nor the navigator.webkitGetUserMedia are not defined, when tested via JavaScript in Bowser.
Is it possible that this is just a compilation issue of some kind or is it just that it's throwing exceptions / errors that I just can't see within Bowser?
You can use the Developer Tools in Safari to debug errors in Bowser (UIWebViews):
https://github.com/EricssonResearch/bowser#debugging-webrtc-scripts
I'm developing this on an iPhone, which does have Inspection enabled, but I don't see any additional options or tools that seem to affect Bowser. Also, Bowser does not run in the simulator, so I can't use any debugging tools there either..
When running the JS in any platform that currently does not support WebRTC (IE, Safari, etc), it does not produce any actual JavaScript errors - it's accurately detected either by my JS (the IF statement above) or, even if I remove that IF statement, it's still cleanly detected by the TokBox JS libraries and informs the user properly.
In Bowser, when I omit my JS detection (the IF statement above) code, it instead produces a JS error or exception that I cannot see, which breaks the page. This implies that the library is likely incompatible with Bowser, but more importantly both navigator.mozGetUserMedia and navigator.webkitGetUserMedia are still "undefined" when I check for them in my detection code. So, if those two methods are "undefined", then it stands to reason that it's still not going to work anyways.
I compiled Bowser from the Git repository and did not modify it, with the exception of setting the developer account in the settings. Is there maybe something missing?
Actually, additional information - that error was due to something unrelated. Now that it's suppressed, the only thing it's telling me is that "This browser is not compatible with WebRTC" from the TokBox platform.
So the getUserMedia API is found now? But TokBox says that the browser is not compatible with WebRTC? Do you know if they do any user agent sniffing?
No, that is not correct - GetUserMedia is not found (it's undefined) - TokBox is displaying that message because it cannot find GetUserMedia.
The error that I mentioned above ("In Bowser, when I omit my JS detection (the IF statement above) code, it instead produces a JS error or exception that I cannot see, which breaks the page.") was unrelated TokBox, it was a bug in my code that I had never seen, because my JS detection code was always in the way of it. When I fixed the error in my JS, the TokBox platform was able to load normally and it agreed with my JS detection code and said that Bowser was not WebRTC compatible (because GetUserMedia was not found).
Looks like the Google getUserMedia test app has moved to http://googlechrome.github.io/webrtc/samples/web/content/getusermedia/gum/
Tested with that sample GUM page above and it does work, however, when I load my page, none of that seems to be defined.
In fact, I added this to the top of all of my other JS includes in
to see if maybe one of them was overriding the function directly, but instead, each of the following alerts "undefine" for each of the alert lines below):<!doctype html>
<html>
<head>
<script type="text/javascript">
alert('before ANY other scripts IN theme.tpl...');
alert(navigator.getUserMedia);
alert(navigator.mozGetUserMedia);
alert(navigator.webkitGetUserMedia);
alert(window.webkitRTCPeerConnection);
alert(window.mozRTCPeerConnection);
</script>
...
Is there something I'm missing here or does it just look like none of this is defined?
Ok, so now that I've ripped apart a non-minified version of opentok.js from TokBox, I can see where it's truly failing:
1) The first failure is on Browser version. It's detecting webkit version 27, but a minimum of webkit 34 is required.
2) It fails when checking the capabilities of webkitRTCPeerConnection, as both of these conditions of this if statement return false:
if (typeof(window.webkitRTCPeerConnection) === 'function' && !!window.webkitRTCPeerConnection.prototype.addStream)
If I had to guess, I'd say that the reason is entirely due to the webkit version.
An chance of upgrading to webkit 34 or higher?
That's easy actually: The UserAgent is set in BowserAppDelegate. It's actually set to 'not Chrome' but I guess that is ignored :-)
Yes, I see that #define KUserAgent @"xxxx" line, but that's not the only issue.
In opentok.js, I actually removed the "return" part of the if statement referenced in point # 1 above so that the function just ignores the version requirement (temporary hack for debugging). However, even ignoring the version deficiency, as it continues with the requirement checks, it then fails on the very next requirement (listed as point # 2 above).
I was only speculating that if the webkit version is actually, physically 27, then perhaps the functionality it's trying to use is not present?
if (typeof(window.webkitRTCPeerConnection) === 'function' && !!window.webkitRTCPeerConnection.prototype.addStream)
This check fails because addStream is not a property of the prototype in our implementation, but instead assigned directly to the peer connection object.
Regarding the webkit version it will be whatever the iOS WebView is using. We are building a part of WebKit as a dependency (JavaScriptCore), but that's only being built for Linux and Android to be used as JavaScript engine.
So then the nature of the check is technically invalid? As in they're assuming it's defined in the prototype, but you're just attaching it to the object directly? If that's the case, I can hack that...
I can see where it's referencing the .addStream() method, but I can't seem to locate where you've attached that method. I'll just manually assign window.webkitRTCPeerConnection.prototype.addStream = [the function], if I can locate said function. How can I reference it?
window.webkitRTCPeerConnection.prototype.addStream = function() {}
should be enough, it'll use the correct function anyway
Adding window.webkitRTCPeerConnection.prototype.addStream = function() {} to the JS before it checks the requirements, basically only allows you to get past the requirement checks. When it tries to load the video chat box, it never actually connects or does anything. I get the normal loader box that it displays before you grant permissions to use the camera/microphone, but it never asks for permissions and it never loads the video stream. Attached a screen shot of what I see in Bowser and also what I see in FireFox or Chrome at the same URL.
I should point out that it IS detecting the number of streams being published and will show multiple video boxes for each browser that's got the page loaded into it, but it's never able to render the video streams inside the boxes.
are you rendering a self-view or only remote streams?
Both - It dynamically detects if new streams are published. In fact, right now, I was testing - I closed all browsers that had it open, opened the test page in Bowser and it had only 1 box (the larger, main box) and then a waited a few seconds. Then, I opened the page on my laptop and started streaming. On my phone, it rendered another (smaller) video box below the main one, as expected. Problem is that neither of them are physically rendering the stream - locally or remotely. It appears that the normal data exchanges and normal logging events are being fired back and forth, as designed, but the rendering of the video streams is just not happening. I'm guessing it's still related to the addStream issue that we've been discussing, but i could be wrong - it could be related to something else that I'm not seeing.
Are you using the latest OpenWebRTC? A couple of important interoperability bugs were fixed about 2 days ago: https://github.com/EricssonResearch/openwebrtc/wiki/Building-OpenWebRTC#update-openwebrtc
Attempting a re-build now, will continue to test/debug. After trapping some errors with try/catch, I can get it to publish a video stream locally, but none of the peer-to-peer or routed connections are working, nor remote streams rendering locally. I'll test after the rebuild and continue to debug. Thanks for the update.
Going back to our previous point of discussion, it appears that none of these have a function called AddStream, either natively or in the prototype, so where is your addStream function?
These are all undefined:
webkitRTCPeerConnection.prototype.addStream RTCPeerConnection.prototype.addStream webkitRTCPeerConnection.addStream RTCPeerConnection.addStream
It's assigned once the object is created, try new webkitRTCPeerConnection({iceServers:[]}).addStream
This is difficult to put in a "small" post, but I'll try.
First, @Rugvip, I hacked the RTCPeerConnection objects to have the addStream function you referenced above:
if(window.webkitRTCPeerConnection) {
var tempHack = new webkitRTCPeerConnection({iceServers:[]});
window.webkitRTCPeerConnection.prototype.addStream = tempHack.addStream;
window.RTCPeerConnection.prototype.addStream = tempHack.addStream;
}
I'm not 100% sure if it worked yet, because I'm iterating through the TokBox/OpenTok code and finding other issues, so this might have solved a problem, but I can't be sure yet.
Prior to debugging further issues, the video and audio stream was rendering locally in Bowser on my iPhone, but it still wasn't publishing to the stream or TokBox. It was notifying the other participants that I connected, but no stream ever came through. So, I started troubleshooting / stepping through some JS errors I was catching via try/catch. I was able to work my through or hack my way through those, but I've come to a point where one particular issue has me stuck.
I started troubleshooting a MediaStreamTrack object issue that was throwing JS errors. MediaStreamTrack objects have an "enabled" property (boolean) that controls whether or not the individual track is enabled. TokBox had the process of updating the enabled property wrapped in a function that they attached to the prototype:
if (window.MediaStreamTrack && typeof(window.MediaStreamTrack) !== 'undefined') {
if (!window.MediaStreamTrack.prototype.setEnabled) {
window.MediaStreamTrack.prototype.setEnabled = function (enabled) {
this.enabled = OT.$.castToBoolean(enabled);
};
}
}
In FireFox and Chrome, window.MediaStreamTrack and window.MediaStreamTrack.prototype exist and this code works, attaching the nearly superfluous function to the prototype.
In Bowser, window.MediaStreamTrack and window.MediaStreamTrack.prototype are not defined in this context, so it does not attach setEnabled to the prototype and it was breaking the JS code.
At some point in their code, it separately calls functions to get the streams of both video and audio and it loops through them and tries to toggle the "enabled" property based on whether or not a provided "muted" variable is set to true or false. Here's the pseudo code:
var videoTracks = webRTCStream.getVideoTracks();
...
var audioTracks = webRTCStream.getAudioTracks();
Then it looped like this:
for (var i=0, num=audioTracks.length; i<num; ++i) {
audioTracks[i].setEnabled(!_muted); }
This thew a JS error when it hit the .setEnabled line, so I wrapped it in a try/catch. This was never setting audioTracks[i].enabled or videoTracks[i].enabled to boolean true, so this left the property as "undefined", but catching the error let the JavaScript was letting me continue past the problem. At that point, it was rendering my audio and video in Bowser, but it was not even attempting to publish to the stream, so other participants could not see or hear me, nor I them. With more troubleshooting, I added some additional code to ensure that it wasn't throwing the exception in the first place:
for (var i=0, num=audioTracks.length; i<num; ++i) {
try {
if(audioTracks[i].setEnabled) {
audioTracks[i].setEnabled(!_muted);
} else {
audioTracks[i].enabled = !_muted;
}
} catch(error) {
alert(error.message);
}
}
So at this point, it's setting the enabled property to true, which it wasn't before, but now the audio and video streams aren't rendering in Bowser anymore (which really doesn't make any sense, logically), but it IS attempting to publish the feed to the stream and throws the following error:
"Publisher PeerConnection Error: The Media resource indicated by the src attribute was not suitable," which is an error code 4 from the window.MediaError.MEDIA_ERR_SRC_NOT_SUPPORTED list.
With the window.MediaStreamTrack and window.MediaStreamTrack.prototype not being defined in Bowser, I'm not sure if there's a problem with my build of if this was done on purpose. This also raises the concern of "what else is not defined the same way between Bowser and Chrome?" and whether or not this difference has anything to do with the issue that I'm working on.
I've done a bunch of reading on various WebRTC implementations recently and I haven't been able to find an answer to the MediaStreamTrack issue.
Also, I've been given permission to throw a copy of our testing/debugging page and scripts on my bluehost instance, so if anyone has a build of Bowser and want's to take a stab at it, maybe that will help. I can upload that tomorrow morning my time (eastern).
As @Rugvip said earlier, we don't add anything to the constructor function prototypes in the JS API, all functions and attributes are added to the object when created.
webkitRTCPeerConnection.prototype.addStream
is undefined
, but
new webkitRTCPeerConnection(config).addStream
is defined.
MediaStreamTrack
doesn't have public constructor (in the standard) so I'm not sure why Chrome and Firefox expose one. You can't use it since it throws "Illegal constructor", but I guess it can be used for monkey patching (like setEnabled() above).
Anyhow, there's nothing wrong with your build.
Alright, good, so the troubleshooting continues and thanks for the verification. With MediaStreamTrack.enabled = true, it's attempting to publish but failing, which is likely related to something in RTCPeerConnection. It could be a similar situation where they're monkey patching and it's failing because of it, but they don't need to or some other type Chrome specific expectation that's in Chrome, but not exactly the same in Bowser.. I'll keep plugging through it.
We really appreciate your efforts, please let us know how we can support further.
One additional request that could help me debug this - do you have a sample page that you know does 2-way video chatting and works in Bowser so that I can test and examine the implementation? In the marketing video, it shows 2 people video chatting, so I'm hoping that maybe you've got that little app hosted or available somewhere. The OpenTok/TokBox platform is massive and it'll be easier to isolate the differences if I had something to which I could compare.
Absolutely: http://demo.openwebrtc.io
We have confirmed it works between all combinations of Bowser, Chrome and Firefox.
When loaded in Bowser, it doesn't let me click Join or Call. When using the "Send link to a friend" url, it loads the page and doesn't load the video chat.
This is really strange, it works for me on a variety of devices. Are you starting Bowser "fresh" before you load a page? I.e. killing it or running from Xcode. There are known issues with releasing resources: https://github.com/EricssonResearch/openwebrtc/issues/38
Would be interesting to know if you can run our self-view example app: https://github.com/EricssonResearch/openwebrtc-examples/tree/master/ios/Selfie
At this point, with a couple of minor tweaks to the opentok.js file, it's displaying the local video again, but it's still not connecting to the opentok servers. I'm getting a specific timeout with their Raptor/Rumor messaging service which sends data back and forth between the browser and their media server, but I've just begun troubleshooting that. This error could be anything at this point.
I can get any page that just simply displays the audio and/or video feed from the current camera/mic in a video tag in the DOM to work fine. Any basic HTML5 sample pages that do that work in my current build of Bowser.
Also, that's correct - I always clear history and kill the app from running before attempting to open it again and load the page. It's cumbersome, but it's the only way it actually reloads the .js and .html content from the server.
As to the demo.openwebrtc.io page, I get the following error text spewing on the page in Bowser now (not sure why the video didn't render locally yesterday when testing it, but it is now):
InvalidStateError: addIceCandidate: no remote description set InvalidSessionSessionDescriptionError: setRemoteDescription: description type "offer" invalid for the current state "have-local-offer" InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set ... (this continues down into the bottom of the background image, where I can no longer read the text)
In my browser on my computer, Chrome in this case, I attempt to "join" and "call", which leaves me with the following error:
Error processing ICE candidate Error processing ICE candidate Error processing ICE candidate Error processing ICE candidate Error processing ICE candidate Error processing ICE candidate ... (this continues down into the bottom of the background image, where I can no longer read the text)
I then tested the page between two different computers using chrome and the page certainly works fine. I'm guessing that the errors I'm getting in both browsers when testing with a Bowser user are Bowser originating. Any thoughts?
Update: I was able to get the two-way video chatting on demo.openwebrtc.io to work on an iPad with iOS 8.1 on it. Are there iOS version limitations?
Nice! The build target is 7.1 currently but I think there is no reason not to only support the latest iOS version. I'm guessing people that would be interested in Bowser will be upgrading their devices.
Did you do any changes? Was this using our demo app or also the one you're working on?
Sorry, I realize now that I'm kind of shifting back and forth talking about what happened with demo.openwebrtc.io and my test page.
The errors that I posted containing "InvalidStateError: addIceCandidate: no remote description set" were the demo.openwebrtc.io page in my 7.1 build on my iPhone.
I tested demo.openwebrtc.io on this iPad with 8.1 and it works properly (so that's good overall, but bad that demo.openwebrtc.io isn't working in Bowser on my iPhone with 7.1).
Right now, I'm testing my test page on the 8.1 iPad and it's not work and throwing errors when attempting to bind the video stream to the video element.
I'm actively troubleshooting my test page on both the 7.1 iPhone and the 8.1 iPad in Bowser, trying to work through the various errors they're throwing (video element stream bind errors in 8.1 and peer connection errors in 7.1).
Currently, I'm unable to confirm whether these issues solely the nature of the JavaScript code, because the demo.openwebrtc.io page did not work on my 7.1 iPhone and my test page with TokBox isn't working in either version. I did post my test page to a publicly accessible place, so at any point, if someone has a build of Bowser and they could possible easily spot the JS issues, I can privately provide the URL.
InvalidStateError: addIceCandidate: no remote description set InvalidSessionSessionDescriptionError: setRemoteDescription: description type "offer" invalid for the current state "have-local-offer"
This (mainly the second error) happens if both clients initiate a call to each other at the same time. The client has generated an offer and installed it locally. But then it gets an offer from the other side. But what it really expected was an answer.
You should only push the call button on one side.
On my 7.1 iPhone and the latest version of Chrome on my laptop:
After around 25 seconds, an empty video panel is rendered in Chrome on my laptop and a few seconds later refreshes with the feed from my iPhone (painfully slow, one frame every 5-8 seconds and both my machines are on WiFi, same network). The initial panel with the feed from my laptop renders in Bowser on my phone, but never refreshes after the first frame. Errors are produced:
1.7 iPhone, same procedure: InvalidStateError: addIceCandidate: no remote description set InvalidStateError: addIceCandidate: no remote description set
Chrome on Laptop: Error processing ICE candidate x 29 (I counted)
Tried between my iPhone and FireFox (latest version) and regardless of whether I initiate the call in FireFox or I initiate the call in Bowser, the other browser see's that a call has been initiated, but no connection is made, no errors are thrown, but the video panels rendered no less than 2 minutes after the call was initiated.
I've tested it between two computers (as a control test) here in the office in varying combinations of FireFox and Chrome (all the latest versions) with complete successand speedy performance.
At this point, demo.opentok.io DOES work in Bowser (even if the performance is unbearable at times), so I'm going to focus solely on troubleshooting my test page that employs TokBox, where I'm still getting video element stream bind errors in 8.1 and peer connection errors in 7.1.
Which iPhone? And are you using current git master of openwebrtc?
I'm not rebuilding the openwebrtc daily, because it takes hours to download and recompile. Last build was a few days ago. When rebuilding openwebrtc, do you have to boostrap, rebuild dependencies and build openwebrtc every time, or can I skip some of that?
iPhone 4 iOS 7.1.2
An iPhone 4 is going to struggle. It can never have hardware codec support and software encoding is rather taxing. However, the settings could be tweaked to make it work better for you (lower resolution and framerate.) I'll put up a patch soon.
As for updating, most of the time normally you would just update openwebrtc itself and rebuild but sometimes and recently we updated dependencies. Check the wiki build page for update instructions.
Yeah, those older iOS devices are going to struggle. The iPhone 4 is ~4.5 year old and I seem to remember that the iPhone 4s was roughly a 2x speed bump. I know that the iPhone 5 works pretty good even at these higher (VGA) resolutions. Perhaps we should put a disclaimer on the Wiki about this?
What's the expected output of the following line in Bowser?
window.URL.createObjectURL(webRTCStreamObj);
Right now, it's returning a "mediastream:xxxxxxxxxxx" formatted url, but the problem is that I've noticed that there's strange looking characters in the string, such as this example:
mediastream:cBb5FS+AhcWq7ty6yT70xaSsFy5kH+fMsNdT
I ran the test and this format is pretty much the same random string of chars. I tested some other versions of FireFox and Chrome, some of which giving me a blob:xxxxx formatted ObjectURL and some giving me the mediastream formatted ObjectURL.
The problem is that after the window.URL.createObjectURL(webRTCStreamObj) output is assigned to the src attribute of the DOM video element, the video element fires the onError function that's been attached to it via videoElement.addEventListener('error', onError, false) code. This fires reporting that the src url is not valid by throwing "MEDIA_ERR_SRC_NOT_SUPPORTED" or code 4.
interface MediaError {
const unsigned short MEDIA_ERR_ABORTED = 1;
const unsigned short MEDIA_ERR_NETWORK = 2;
const unsigned short MEDIA_ERR_DECODE = 3;
const unsigned short MEDIA_ERR_SRC_NOT_SUPPORTED = 4;
readonly attribute unsigned short code;
};
So, the question is - - Why would a videoElement.src = window.URL.createObjectURL(webRtcStream); call throw that error?
Additional notes:
webRTCStreamObj is of type MediaStream (appears to be native) window.URL.createObjectURL is native videoElement.src the output of a var videoElement = document.createElement('video'); call (this is standard/native).
This LOOKS exactly the same as the way demo.openwebrtc.io does it, but that page does not throw this error. I'm using the most recently available code for both bower and openwebrtc.
I'd be thrilled if you took the time to document your findings on this page:
https://github.com/EricssonResearch/bowser/wiki/Developing-for-Bowser
It will help other developers to get started with Bowser. The wiki is currently a bit thin, we plan to add more ourselves. Thanks.
Honestly, I've changed very little thus far. In Bowser, I've only changed the version that's being reported in the header of the Bowser source. In openwebrtc, I've changed nothing. For developers writing apps/web pages that are meant to run in Bowser, there's a few things to document in regards to the differences between the prototypes, method names, and any other little quirks that might deviate slightly from the other two main WebRTC enabled browsers. I'd be happy to circle back and write up some notes or things to consider when building for Bowser once I'm finished getting the TokBox/OpenTok platform to work on it.
On that note, have you considered setting the actual user-agent to be unique, such as "Bowser", rather than "not Chrome" as it is now? I know this basically makes it compatible with chrome-specific code (which it should be by default), but since there are significant differences in the ostensible implementation of the common WebRTC JS functions, perhaps it would be nice to be able to more definitively identify it as Bowser? Just a thought.
Also, does anyone have any suggestions or thoughts on my previous two posts (they're really a single post, I just posted some additional notes after my original posting)? I've been stuck for several days at that point and have been forced to jump over to some other little projects temporarily due to the roadblock.
After compiling and running it on my iPhone, I attempted to load a page that I've been working on that uses the OpenTok/TokBox libraries (WebRTC-based video chat platform).
The page loads with no errors, but the pre-detection of whether or not WebRTC is present fails. The variables/objects that it's looking for are below:
If I remove the pre-detection, I get a JS error that prevents the page from fully loading.
I'm assuming that perhaps the functions are implemented differently in Bowser, so I have the following questions about the WebRTC implementation in Bowser:
1) Were navigator.mozGetUserMedia or nagivator.webkitGetUserMedia implemented under different object/method names?
2) Is there any documentation regarding the potential differences in building web pages for Bowser?
3) Where in the source would the definition of the methods attached to the nagivator object be found?
It's possible that TokBox/OpenTok would not be compatible with your WebRTC implementation or that perhaps only some minor shims would be required to use the correct object/method names - I'm just trying to determine in which direction I should go.