Sound problems with org.apache.cordova.plugin.media

[geoffreybauduin]

Hi,

we've been discussing about this problem with @songz using mails, and he suggested to post this issue here.

The problem is really simple: the first time we initialize our layouts for a video conference, it works well. The second time we initialize them (the DOM has changed a lot during these two conferences), some errors are being thrown, but only on Xcode console (which is a bit annoying, considering we're registering an exception handler TB.on("exception", ...);)

Here is how we initialize the publisher:

$scope.publisher = TB.initPublisher($scope.apiKey, "layoutContainer",
  width: 0
  height: 0
  publishAudio: true
  publishVideo: true
)

and here how we initialize the subscriber

subscriberProperties =
  insertMode: "replace"
  subscribeToAudio: true
  subscribeToVideo: true
  audioVolume: 100
$scope.session.subscribe event.stream, "layoutContainer2", subscriberProperties, ->
  console.log "Subscribed"

This errors occurs since iOS 8.1. The Webview is even bouncing with some red stuff on top !

[songz]

I think there are some problems here. InsertMode is not yet implemented. TB.on exception is also not yet implemented, I'll file issues for them. https://github.com/songz/cordova-plugin-opentok/issues/127 https://github.com/songz/cordova-plugin-opentok/issues/128

[songz]

What is the audio issue, and how can I reproduce it? from your code, it looks like you are using angularJS. Does the audio issue occur in a normal app as well, or is it only in angularJS?

[geoffreybauduin]

Well this problem cannot be reproduced everytime... It's happening sometimes, we don't really know why or when, the main fact being it happens after the second time we launch a video session (as we told you by mail).

The issue is occuring on AngularJS, and we haven't been investigating that much to see if it was occuring in a normal app as well.

[aullman]

I'm just getting caught up here, what is the audio issue exactly? Any audio issue sounds like it would probably be an issue with the OpenTok SDK rather than Cordova right?

[tuttiflirty]

Hello Adam,

thanks a lot for you answer.

We have the sound when app is freshly installed and first session is done. Then it works only 20-30% of cases. We use cordova implementation.

We described our problem with details here: https://github.com/songz/cordova-plugin-opentok/issues/118

Song has already logger some issues, that could probably solve the problem..

Song says: I think there are some problems here. InsertMode is not yet implemented. TB.on exception is also not yet implemented, I'll file issues for them.

127 https://github.com/songz/cordova-plugin-opentok/issues/127

128 https://github.com/songz/cordova-plugin-opentok/issues/128

Thank you, Peter

On Fri Nov 28 2014 at 00:08:39 Adam Ullman notifications@github.com wrote:

I'm just getting caught up here, what is the audio issue exactly? Any audio issue sounds like it would probably be an issue with the OpenTok SDK rather than Cordova right?

— Reply to this email directly or view it on GitHub https://github.com/songz/cordova-plugin-opentok/issues/118#issuecomment-64836562 .

[aullman]

I'm still not clear what the audio problem is though. You seem to describe an issue with DOM layout, not with sound. The missing 'exception' handler just explains why you're not getting errors in JS. The insertMode will just be ignored and you're using the default 'replace' anyway. So those things shouldn't fix your problem.

So is there a sound problem? If so, what is the sound problem?

Is there a DOM problem? If so can you give us a screenshot of what you mean? Also any errors you're receiving in the JS console or in XCode would also be helpful.

My guess as to an issue is that you might not be recreating "layoutContainer" and "layoutContainer2" when you initialise the layout the second time. The Publisher and Subscribers will replace those elements so the second time they won't be there anymore to be replaced.

[geoffreybauduin]

Well, we're using AngularJS and these two divs are in our template, so I guess it's not really this kind of problem. The main problem here is that the two phones are not playing any sound the second time. (You can see our code there #111 ).

Here are the logs we can read on Xcode:

2014-12-01 11:34:24.131 Tilk[9174:3242150] JS: TB Exception Handler added
2014-12-01 11:34:24.144 Tilk[9174:3242150] iOS Connecting to Session
2014-12-01 11:34:24.251 Tilk[9174:3242150] THREAD WARNING: ['OpenTokPlugin'] took '107.185059' ms. Plugin should use a background thread.
2014-12-01 11:34:24.256 Tilk[9174:3242150] success
2014-12-01 11:34:27.278 Tilk[9174:3242150] iOS Connected to Session
2014-12-01 11:34:27.279 Tilk[9174:3242150] iOS creating data from stream: <OTConnection: 0x19de5f30>
2014-12-01 11:34:27.279 Tilk[9174:3242150] object for session is {
    sessionConnectionStatus = OTSessionConnectionStatusConnected;
    sessionId = "1_MX40NTA1NjYyMn5-MTQxNzQzMDA2MTE5OH5SY29PRzJ1QmhNSVdEY2J5ZWZTMDR5L2F-UH4";
}
2014-12-01 11:34:27.287 Tilk[9174:3242150] JS Lib: session event received -  [object Object]
2014-12-01 11:34:27.287 Tilk[9174:3242150] JS Lib: sessionConnectedHandler -  [object Object]
2014-12-01 11:34:27.288 Tilk[9174:3242150] iOS creating data from stream: <OTConnection: 0x19de63a0>
2014-12-01 11:34:27.293 Tilk[9174:3242150] JS Lib: session event received -  [object Object]
2014-12-01 11:34:27.389 Tilk[9174:3242150] domId exists but properties width or height is not specified
2014-12-01 11:34:27.389 Tilk[9174:3242150]  width: 80 and height: 80 for domId layoutContainer, and top: 284, left: 230
2014-12-01 11:34:27.390 Tilk[9174:3242150] JS Lib: creating publisher -  [object Object]
2014-12-01 11:34:27.390 Tilk[9174:3242150] 5
2014-12-01 11:34:27.391 Tilk[9174:3242150] JS: Objects being updated in TBUpdateObjects
2014-12-01 11:34:27.391 Tilk[9174:3242150] JS: Object updated
2014-12-01 11:34:27.392 Tilk[9174:3242150] JS sessionId: TBPublisher
2014-12-01 11:34:27.392 Tilk[9174:3242150] 5
2014-12-01 11:34:27.392 Tilk[9174:3242150] The Width is: 80
2014-12-01 11:34:27.397 Tilk[9174:3242150] iOS creating Publisher
2014-12-01 11:34:28.709 Tilk[9174:3242150] THREAD WARNING: ['OpenTokPlugin'] took '1311.717041' ms. Plugin should use a background thread.
2014-12-01 11:34:28.712 Tilk[9174:3242150] iOS Publish stream to session
2014-12-01 11:34:28.716 Tilk[9174:3242150] success
2014-12-01 11:34:28.716 Tilk[9174:3242150] success
2014-12-01 11:34:28.717 Tilk[9174:3242150] success
2014-12-01 11:34:29.388 Tilk[9174:3242150] JS Lib: publisher event received -  [object Object]
2014-12-01 11:34:29.389 Tilk[9174:3242150] JS Lib: publisher streamCreatedHandler -  [object Object]
2014-12-01 11:34:29.389 Tilk[9174:3242150] JS Lib: publisher streamCreatedHandler -  [object Object]
2014-12-01 11:34:29.390 Tilk[9174:3242150] JS Lib: publisher streamCreatedHandler -  
2014-12-01 11:34:37.050 Tilk[9174:3242150] iOS Received Stream
2014-12-01 11:34:37.056 Tilk[9174:3242150] JS Lib: session event received -  [object Object]
2014-12-01 11:34:37.056 Tilk[9174:3242150] JS Lib: streamCreatedHandler -  [object Object]
2014-12-01 11:34:37.169 Tilk[9174:3242150] JS Lib: creating subscriber -  [object Object]
2014-12-01 11:34:37.170 Tilk[9174:3242150] 3
2014-12-01 11:34:37.170 Tilk[9174:3242150] JS Lib: final subscriber position -  [object Object]
2014-12-01 11:34:37.171 Tilk[9174:3242150] iOS subscribing to stream
2014-12-01 11:34:37.223 Tilk[9174:3242150] THREAD WARNING: ['OpenTokPlugin'] took '52.481934' ms. Plugin should use a background thread.
2014-12-01 11:34:37.232 Tilk[9174:3242150] success
2014-12-01 11:34:39.170 Tilk[9174:3242799] 11:34:39.170 ERROR:     [0x8039000] 1230: AUIOClient_StartIO failed (-66637)
2014-12-01 11:34:39.171 Tilk[9174:3242799] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥
2014-12-01 11:34:39.211 Tilk[9174:3242957] 11:34:39.211 ERROR:     [AURemoteIO::IOThread] >aurioc> 1497: AURemoteIO@0x1530a620: IOThread exiting with error 0x10004006
2014-12-01 11:34:39.863 Tilk[9174:3242799] 11:34:39.863 ERROR:     [0x8039000] 1230: AUIOClient_StartIO failed (-66637)
2014-12-01 11:34:39.864 Tilk[9174:3242799] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥
2014-12-01 11:34:40.508 Tilk[9174:3242970] 11:34:40.508 ERROR:     [AURemoteIO::IOThread] >aurioc> 1497: AURemoteIO@0x1530a620: IOThread exiting with error 0x10004006
2014-12-01 11:34:41.155 Tilk[9174:3242799] 11:34:41.155 ERROR:     [0x8039000] 1230: AUIOClient_StartIO failed (-66637)
2014-12-01 11:34:41.165 Tilk[9174:3242799] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥
2014-12-01 11:34:41.412 Tilk[9174:3242980] 11:34:41.412 ERROR:     [AURemoteIO::IOThread] >aurioc> 1497: AURemoteIO@0x1530a620: IOThread exiting with error 0x10004006
2014-12-01 11:34:42.054 Tilk[9174:3242799] 11:34:42.054 ERROR:     [0x8039000] 1230: AUIOClient_StartIO failed (-66637)
2014-12-01 11:34:42.054 Tilk[9174:3242799] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥
2014-12-01 11:34:42.056 Tilk[9174:3242986] 11:34:42.056 ERROR:     [AURemoteIO::IOThread] >aurioc> 1497: AURemoteIO@0x1530a620: IOThread exiting with error 0x10004006
2014-12-01 11:34:42.703 Tilk[9174:3242799] 11:34:42.703 ERROR:     [0x8039000] 1230: AUIOClient_StartIO failed (-66637)
2014-12-01 11:34:42.709 Tilk[9174:3242799] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥
2014-12-01 11:34:44.897 Tilk[9174:3242989] 11:34:44.897 ERROR:     [AURemoteIO::IOThread] >aurioc> 1497: AURemoteIO@0x1530a620: IOThread exiting with error 0x10004006
2014-12-01 11:34:44.997 Tilk[9174:3242150] iOS Connected To Stream
2014-12-01 11:34:45.008 Tilk[9174:3242150] JS Lib: session event received -  [object Object]
2014-12-01 11:34:45.551 Tilk[9174:3242799] 11:34:45.551 ERROR:     [0x8039000] 1230: AUIOClient_StartIO failed (-66637)
2014-12-01 11:34:45.556 Tilk[9174:3242799] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥

Please note that this problem is happening the second time we load the video view, and never the first time. If we close or kill the app, it'll work again, but only the first time.

[aullman]

These look like related issues:

https://forums.tokbox.com/ios/opentok-missing-audio-in-audio-video-conference-session-t44061 https://forums.tokbox.com/bugs/missing-audio-in-audio-video-conference-session-t44062 http://stackoverflow.com/questions/26433988/opentok-missing-audio-in-audio-video-conference-session

It seems like we're not letting go of the audio device correctly when we disconnect from the session. I don't know much about iOS though. @songz do you have any idea?

[geoffreybauduin]

After a quick look at these issues, it seems that the problem reported on StackOverflow (third link) hasn't been solved by the author yet. Could it be something related to the iOS SDK itself and not the Cordova Plugin? Even if it would be strange since it's widely used and only a few people seem to have issues with that.

We've been experimenting through this problem right there: https://github.com/lght/test-opentok-cordova . master branch is not using org.apache.cordova.media, and the branch with-media is using cordova.plugin.media. While we are on branch master, there's nothing wrong with the sound... But if we use the second branch, the second time we launch the conversation, we can see the same error in our Xcode console: 2014-12-04 15:44:53.383 HelloCordova[2102:959847] 15:44:53.383 ERROR: [0x5c87000] 1230: AUIOClient_StartIO failed (-66637) 2014-12-04 15:44:53.389 HelloCordova[2102:959847] ERROR[OpenTok]:Audio device error: AUGraphStart error: ˇ˛˚≥. It seems to be a problem related to org.apache.cordova.media

[aullman]

Yes, it looks like an OpenTok issue but possibly one that the Cordova plugin will need to work around. I'm guessing we're not releasing the audio device properly the first time which is causing the error.

[cnavarro1109]

Is there an update/patch to resolve this issue? At the moment, I get it even when I run the songz DEMO application from github for the first time. On the iPhone (Latest build), I get video but no sound from the connected PC (Browser). It works find when I use 2 Browser based connections but not with the iPhone. Overall, sending and receiving video works but the sound is not coming in through the iPhone.

[geoffreybauduin]

@cnavarro1109 this issue is more about the conflict between this plugin and org.apache.cordova.media. You should open a new issue

[devand123]

Any more insight to this? Is there a workaround where we can release the audio device so the two plugins can play nicely together?

[geoffreybauduin]

Not really, we've been solving this issue by removing the "extra-sound" from our app. We haven't heard yet from them so I don't really know if it's being handled or not...