Please provide example on how to mix audioMic and audioApp samples in ReplayKitExample BroadcastExtension SampleHandler

[inertjade]

Description

Room participants cannot hear app audio along with mic audio of the presenter sharing his device screen using ReplayKit.

Steps to Reproduce

1. Run Twilio Video ReplayKit Example on iPhone A and tap on "Start Broadcast"
2. Run VideoQuickStart on iPhone B and join the same room as Step 1.
3. Note that User on iPhone B can listen in to mic on iPhone A.
4. iPhone A plays a recorded video with sound.
5. iPhone B cannot listen to the sound of the recorded video, playing on iPhone A.

Expected Behavior

User on iPhone B can listen to the app audio and the mic audio of iPhone A.

Actual Behavior

User on iPhone B can hear nothing when an app audio plays on iPhone A.

Reproduces How Often

100%

Versions

All relevant version information for the issue.

Video iOS SDK

3.4.1 via CocoaPods

Xcode

11.5

iOS Version

13.5

iOS Device

iPhone 11

[inertjade]

Hi @ceaglest , You may be aware of this already. Agora, another vendor of Video services, has a similar demo app that uses ReplayKit Framework for screen sharing on iOS Devices. That demo app mixes both mic and app Audio streams. Here is the link to the demo app.

Demo App https://github.com/AgoraIO/Advanced-Video/tree/master/iOS&macOS/Agora-Screen-Sharing

Another helpful link. https://docs.agora.io/en/Video/audio_effect_mixing_apple?platform=iOS

[ceaglest]

Hi @inertjade,

Thanks for the request! Demonstrating how to publish mixed app and mic audio is something that we would like to do in ReplayKitExample. In this case I think it could be accomplished either with:

1. An AudioUnit of type kAudioUnitSubType_MultiChannelMixer, which takes the mic and app audio and mixes them into a single stream.
2. AVAudioEngine's mixer and manual input. This would be Apple's preferred approach on the latest iOS releases.
3. Manually mixing the audio waveforms (since the sample rates are the same at 44.1 kHz)

The solution should be able to mix buffers of different sizes because ReplayKit may not always deliver 1024 samples at a time. I will ask the team about scheduling the example work in an upcoming sprint, but no ETA as of yet.

Best, Chris

[inertjade]

Hi @ceaglest ,

Does any of the 3 different methods you mentioned, support muting, or lowering volume level of either of the mic and app audio? Just that, having the option to mute/lower volume at runtime would be extremely favorable?

[ceaglest]

Adjusting the volume of each input would be possible with any of the 3 solutions.

[inertjade]

@ceaglest Just checking in. Have you heard back from your team about scheduling the example work?

[inertjade]

@ceaglest Just checking in, if any progress has been made on this topic.

[inertjade]

@ceaglest @piyushtank Just checking in. Do you have any updates on this topic?

[piyushtank]

@inertjade We do not have audio mixing in ReplayKit example planed in coming sprints but to let you know that we are working on ReplayKit features where the VideoSDK will provide the ReplayKit support out of the box - that means less code developers need to write to support screen sharing in an app. I have created a ticket for your audio mixing request in that epic, however its not a priority right now.

Let us know if you have any questions.

[Fanpei920]

Is there any updates on this issue?

[inertjade]

@Fanpei920 No updates. I don't think Twilio is interested in this feature. Agora has had it since July 2020.

[piyushtank]

@Fanpei920 @inertjade We have made improvements in our Video SDK support for ReplayKit over the past few months: the In App Screen capture is supported by Video SDK APIs out of the box now. See changelogs here. And, internally, we have made great progress in supporting the broadcast system capture the same way, and we plan to ship APIs for it in the future.

We acknowledge the enhancement requested in this ticket, and we have a ticket in our backlog to implement the enhancement. At present, our team is busy working on other high-priority items. We will come back to it when we get a chance in the future.

[Fanpei920]

@piyushtank

Cool :+1: I'll try that out of box API, Thanks a lot!

[piyushtank]

@Fanpei920 thanks! Try it out and let us know if you have any feedbacks.

[Fanpei920]

@piyushtank

It seems that not quite fit my requirement. Is there a way of implementing mix audioApp and audioMic in inApp capture (use RPScreenRecorder)

here is the sample code .

  recorder.startCapture(handler: { (sampleBuffer, type, error) in
      if error != nil {
          print("Capture error: ", error as Any)
          return
      }

      switch type {
          case RPSampleBufferType.video:
              self.videoSource?.processFrame(sampleBuffer: sampleBuffer)
              break
          case RPSampleBufferType.audioApp:
              // <- audioApp is captured here.
              break
          case RPSampleBufferType.audioMic:
              break
          default:
              break
      }

Would you like to provide a example or some idea in implementing this feature? I really need this feature, thanks!