Not able to `consumeFrame` from iOS Broadcast Extension

[karolkulesza]

I've successfully incorporated the latest version (v. 2.15.1.6758-ios) of OpenTok.framework into my test iOS Broadcast Extension (extension uses ReplayKit 2.0 and RPBroadcastSampleHandler class: https://developer.apple.com/documentation/replaykit/rpbroadcastsamplehandler).

However, any attempts to send any frames (not only the ones generated iOS system broadcast stream, but any frame, even the ones generated using static images from local resources) to the other testing device, using OpenTok SDK, fail, due to black view in the publisher view on the other device.

In the broadcast extension, I have used the sample code for implementing a custom video capturer (conforming to OpenTok's OTVideoCapture protocol), provided by OpenTok: https://github.com/opentok/opentok-ios-sdk-samples/tree/master/Screen-Sharing .

One quite important note here: After creating a custom video capturer, I was able to transfer frames manually to the receiving device, BUT it's working only in the iOS app. Exactly the same code does not work when executed in iOS app extension (although OpenTok session is properly initialized and working as expected). I've also debugged both situations and seems like the arguments' values passed to OpenTok SDK prior sending the frame are the same (between iOS app execution and iOS app extension execution), so it clearly suggests an issue in OpenTok SDK in the scenario of invoking its APIs from iOS app extension (specifically: videoCaptureConsumer.consumeFrame(:) API).

Question: Has OpenTok.framework been tested in the scenario of running within iOS Broadcast Extension? If yes, can you please provide any working sample code?

[bsrao]

@karolkulesza Is it possible to provide a sample app that demonstrate the issue ? so that I can check what is going on.

[karolkulesza]

@bsrao :

Hi Sridhar,

Yes, I've just shared with you (via @) a test empty app with iOS Broadcast Extension that connects with TokBox (directly from this extension) and tries to send a sample image frame (from local resources). I've used the sample code provided OpenTok, for screen sharing scenario (see TBScreenCapture class, where instead of screenshot of the app, I'm using a static image from local resources).

Before running it, please update "sessionId" and "token" values with valid ones, in class SampleHandler, in BroadcastExtension target.

After installing on iOS device, you won't see anything in the app itself, you have to invoke iOS broadcasting feature on iOS12. In order to do that, on iOS12 device please enable "Screen Recording" feature to be visible in iOS Control Center (via Settings -> Control Center -> Customize Controls) and then start broadcasting by showing Control Center, 3D tapping on broadcast/record icon and selecting "BroadcastExtension" from the list (as indicated in the screenshot attached here).

Optionally, if you'd like to debug this Broadcast Extension (as it is a separate process on iOS), you can use "Debug -> Attach to Process" feature in Xcode.

Let me know please in case of any issues in reproducing it.

Regards, Karol

[MathiasDum]

I'm having the the same issue

[bsrao]

@MathiasDum @karolkulesza I am able to screen share from the extension using iOS 2.11.3 but looks like there is an issue with 2.15. Please try 2.11.3 for now, meanwhile we will look into the issue with 2.15.

[etown]

Are you able to also process the app audio from the broadcast extension?

[karolkulesza]

@etown : Yes, please take a look at one of the methods declared by RPBroadcastSampleHandler protocol:

[etown]

@karolkulesza Thank you, I am aware replaykit provides the frames, but I’m looking to see in example of consuming the audio with opentok. Thank you!

[etown]

@karolkulesza could you please share the sample extension project you had before? it would be extremely helpful. thank you!

[etown]

@bsrao can you confirm if the issue is currently affecting 2.15? also, could you provide replaykit sample code? thank you!

[bsrao]

@etown Yes, it is effecting 2.15. I don't have any sample code that I can share but it will be straight forward, as @karolkulesza shared, you need to grab the frame from processSampleBuffer and consume it using external capturer (you can use custom video driver class in the sample repo).

[msach22]

Hi Folks, we don't have any sample code for this yet, but we are working on fix that should be released with 2.16.0.

[bsrao]

This is an almost finished working sample for Broadcast extension using OpenTok SDK : https://github.com/opentok/opentok-ios-sdk-samples/tree/broadcast-ext-sample Feel free to share any issues you find!

[TheNasirKhan]

I have implemented the above extension in my project but it doesn't send any frame to the other side, even i have tried it directly from this project. it doesn't appear other side of a call even though a session is being created, frames are available it doesn't even add a subscriber to the other side.

It always post error in didFailWithError, with an error "Timed out while attempting to publish."

[v-kpheng]

Apologies for the belated reply. The Timed out while attempting to publish error suggests that frames aren't being sent. There may be multiple reasons for this but unlikely to be related to the SDK. Can you please see if you can reproduce using the updated version: https://github.com/opentok/opentok-ios-sdk-samples/tree/main/Broadcast-Ext? Also, the broadcast sample app only works for H264.