OTPublisher never be published with internal error from AudioUnitInitialize

[hieultse03355]

Hello everyone, have a nice day! First, I want to thank you guys for provides great cloud communications and VoIP services. Im an iOS developer and using your service for my system for years.

Rencently, I have encountered an issue about toxbok iOS SDK. I was using your CallKitDemo (https://github.com/opentok/opentok-ios-sdk-samples-swift/tree/main/CallKit/CallKitDemo) (version toxbok 2.25.0, iOS 16.4).

when starting call, SDK will invoke startCapture and startRendering in OTDefaultAudioDevice, and sometimes when trying setupAudioUnit with this func AudioUnitInitialize then it throws error like this:

AURemoteIO.cpp:1151 failed: 268451843 (enable 2, outf< 1 ch, 48000 Hz, Int16> inf< 1 ch, 0 Hz, Float32>) 2023-05-10 13:28:01.242442+0900 CallKitDemo[736:69369] ERROR[OpenTok]:Audio device error: setupAudioUnit.AudioUnitInitialize returned error: 268451843 2023-05-10 13:28:01.249726+0900 CallKitDemo[736:69431] [aurioc] AURemoteIO.cpp:1761 error -308 from GetCurrentFormats

And OTPublisher will never be published because of this error, then CallKit will end up this call automatically. This error will be reproduced 100% percent when you use an Airpod and connect it to device during the Call because of AVAudioSessionRouteChangeNotification will re-invoke startCapture and startRendering to Restart the audio units. This error only occurred when using with CallKit.

Please give me any idea for this situation. Thank you guys so much.

[jvalli]

Hi @hieultse03355, the error 268451843 is a timeout. Can you provide more details about the flow of your app?

• Were you on OT call and did you receive a real phone call? Or were you on a real phone call and want to switch to OT call?
• How do you integrate CallKit?

It's not clear for me how to reproduce the issue.

[hieultse03355]

1. Were you on OT call and did you receive a real phone call? Or were you on a real phone call and want to switch to OT call?

-> I was on OT call only.

2. How do you integrate CallKit?

-> I was using your setting CallKit flow in your CallKitDemo for my app, and the issue I was reproducing by using your CallKitDemo.

You just open your CallKitDemo then run and do steps below then the issue will be reproduced:

• Step 1: join 2 devices Device 1 (iOS 16.0), Device 2 (iOS 16.4) into a session:

• Step 2: connect an AirPods to Device 2:

• Step 3: When you connected AirPods, AVAudioSessionRouteChangeNotification will be notified and invoke 2 func startCapture and startRendering to restart Audio Units. Then check out terminal for log and see the error and shut down your CallKit:

This issue will be reproduced when your app or toxbok SDK invoke 2 func startCapture and startRendering in very random case not only case Connecting to an AirPods and throws error at AudioUnitInitialize. @jvalli thank you for supporting me

[jvalli]

Hi @hieultse03355, I just did a quick check and figure out that the sample app uses a custom audio driver: https://github.com/opentok/opentok-ios-sdk-samples-swift/blob/main/CallKit/CallKitDemo/OTDefaultAudioDevice.m this one seems to out of date, not sure about it.

Anyway can you try to restart audio units when AVAudioSessionMediaServicesWereResetNotification is received: https://github.com/opentok/opentok-ios-sdk-samples-swift/blob/main/CallKit/CallKitDemo/OTDefaultAudioDevice.m#L727&L747.

We will try to update custom audio driver sample apps soon.

[v-kpheng]

We're tracking this internally: https://jira.vonage.com/browse/VIDCS-898.

[hieultse03355]

Hi guys, sorry for late reply,

I tried to restart audio units when AVAudioSessionMediaServicesWereResetNotification is received as you said but this issue still persists. i think the key here is not WHEN or WHERE you try to restart Audio Units. it is about how you set up Audio Units. As i told before, this issue will be reproduced in very random case when you try to be making a call not only the case when you try to be making Connecting to an Airpods or any kind of device . And it occurred either inBackground or inActive when you were using CallKit for a call.

The flow look like this, please give it a look:

- startCapture

- startRendering

and in random case, 2 these function could not set up Audio Units for unknown reason which i never know what doSubcribe and doPublish did inside the SDK.

The Error throws at Step 5 exactly the same in these function (- startCapture, - startRendering):

AURemoteIO.cpp:1151 failed: 268451843 (enable 2, outf< 1 ch, 48000 Hz, Int16> inf< 1 ch, 0 Hz, Float32>) 2023-05-10 13:28:01.242442+0900 CallKitDemo[736:69369] ERROR[OpenTok]:Audio device error: setupAudioUnit.AudioUnitInitialize returned error: 268451843 2023-05-10 13:28:01.249726+0900 CallKitDemo[736:69431] [aurioc] AURemoteIO.cpp:1761 error -308 from GetCurrentFormats

Please give me more insight about this situation, thank you guys so much.

By the way i can not access to this (https://jira.vonage.com/browse/VIDCS-898 < Unable to sign in)

@v-kpheng @jvalli

[jvalli]

Hi @hieultse03355, if you are working with the sample app CallKitDemo, the app is not running with the default audio driver from the iOS SDK because the demo app overrides audio driver and uses it's own: https://github.com/opentok/opentok-ios-sdk-samples-swift/blob/main/CallKit/CallKitDemo/SpeakerboxCall.swift#L109 So the AudioUnitInitialize that fails is not part of the iOS SDK, is part of the demo app, as your screenshot specifies: https://github.com/opentok/opentok-ios-sdk-samples-swift/blob/main/CallKit/CallKitDemo/OTDefaultAudioDevice.m#L1157

You can try to do not use the custom audio driver from CallKitDemo or try this workaround on the customized audio driver: on - (BOOL)setupAudioUnit:(AudioUnit *)voice_unit playout:(BOOL)isPlayout.

`

...
result = AudioUnitInitialize(*voice_unit);

// This patch is pickedup from WebRTC audio implementation and
// is kind of a workaround. We encountered AudioUnitInitialize
// failure in iOS 13 with Callkit while switching calls. The failure
// code is not public so we can't do much.
int failed_initalize_attempts = 0;
int kMaxInitalizeAttempts = 5;
while (result != noErr) {
    ++failed_initalize_attempts;
    if (failed_initalize_attempts == kMaxInitalizeAttempts) {
        // Max number of initialization attempts exceeded, hence abort.
        return false;
    }
    [NSThread sleepForTimeInterval:0.1f];
    result = AudioUnitInitialize(*voice_unit);
}

if (CheckError(result, @"setupAudioUnit.AudioUnitInitialize")) {
    return NO;
}

`

[v-kpheng]

@hieultse03355, the jira link can only be accessed by employees. Sorry.

[hieultse03355]

Hi @jvalli @v-kpheng

unfortunately, 2 of your workaround could not help,

• if i do not override the audio driver, tokbox SDK still throws this issue with exactly the same error logs
• i also try your above code to re-try AudioUnitInitialize after it threw the error but this issue also remains the same after 5 times of re-try

i knows you guys can not do much,

but for the last, please confirm about:

• are the ways you guys set up AudioUnitSetProperty and stream_format AudioStreamBasicDescription still correct? and have these based on any formula?
• are there documentations about these set up ?

OSStatus result;

    mach_timebase_info(&info);

    UInt32 bytesPerSample = sizeof(SInt16);
    stream_format.mFormatID    = kAudioFormatLinearPCM;
    stream_format.mFormatFlags =
    kLinearPCMFormatFlagIsSignedInteger | kAudioFormatFlagIsPacked;
    stream_format.mBytesPerPacket  = bytesPerSample;
    stream_format.mFramesPerPacket = 1;
    stream_format.mBytesPerFrame   = bytesPerSample;
    stream_format.mChannelsPerFrame= 1;
    stream_format.mBitsPerChannel  = 8 * bytesPerSample;
    stream_format.mSampleRate = (Float64) kSampleRate;

    AudioComponentDescription audio_unit_description;
    audio_unit_description.componentType = kAudioUnitType_Output;

    audio_unit_description.componentSubType = kAudioUnitSubType_VoiceProcessingIO;

    audio_unit_description.componentManufacturer = kAudioUnitManufacturer_Apple;
    audio_unit_description.componentFlags = 0;
    audio_unit_description.componentFlagsMask = 0;

    AudioComponent found_vpio_unit_ref =
    AudioComponentFindNext(NULL, &audio_unit_description);

    result = AudioComponentInstanceNew(found_vpio_unit_ref, voice_unit);

    if (CheckError(result, @"setupAudioUnit.AudioComponentInstanceNew")) {
        return NO;
    }

    if (!isPlayout)
    {
        UInt32 enable_input = 1;
        AudioUnitSetProperty(*voice_unit, kAudioOutputUnitProperty_EnableIO,
                             kAudioUnitScope_Input, kInputBus, &enable_input,
                             sizeof(enable_input));
        AudioUnitSetProperty(*voice_unit, kAudioUnitProperty_StreamFormat,
                             kAudioUnitScope_Output, kInputBus,
                             &stream_format, sizeof (stream_format));
        AURenderCallbackStruct input_callback;
        input_callback.inputProc = recording_cb;
        input_callback.inputProcRefCon = (__bridge void *)(self);

        AudioUnitSetProperty(*voice_unit,
                             kAudioOutputUnitProperty_SetInputCallback,
                             kAudioUnitScope_Global, kInputBus, &input_callback,
                             sizeof(input_callback));
        UInt32 flag = 0;
        AudioUnitSetProperty(*voice_unit, kAudioUnitProperty_ShouldAllocateBuffer,
                             kAudioUnitScope_Output, kInputBus, &flag,
                             sizeof(flag));
        // Disable Output on record
        UInt32 enable_output = 0;
        AudioUnitSetProperty(*voice_unit, kAudioOutputUnitProperty_EnableIO,
                             kAudioUnitScope_Output, kOutputBus, &enable_output,
                             sizeof(enable_output));

    } else
    {
        UInt32 enable_output = 1;
        AudioUnitSetProperty(*voice_unit, kAudioOutputUnitProperty_EnableIO,
                             kAudioUnitScope_Output, kOutputBus, &enable_output,
                             sizeof(enable_output));
        AudioUnitSetProperty(*voice_unit, kAudioUnitProperty_StreamFormat,
                             kAudioUnitScope_Input, kOutputBus,
                             &stream_format, sizeof (stream_format));
        // Disable Input on playout
        UInt32 enable_input = 0;
        AudioUnitSetProperty(*voice_unit, kAudioOutputUnitProperty_EnableIO,
                             kAudioUnitScope_Input, kInputBus, &enable_input,
                             sizeof(enable_input));
        [self setPlayOutRenderCallback:*voice_unit];
    }

    Float64 f64 = 0;
    UInt32 size = sizeof(f64);
    OSStatus latency_result = AudioUnitGetProperty(*voice_unit,
                                                   kAudioUnitProperty_Latency,
                                                   kAudioUnitScope_Global,
                                                   0, &f64, &size);
    if (!isPlayout)
    {
        _recording_AudioUnitProperty_Latency = (0 == latency_result) ? f64 : 0;
    }
    else
    {
        _playout_AudioUnitProperty_Latency = (0 == latency_result) ? f64 : 0;
    }

    // Initialize the Voice-Processing I/O unit instance.
    result = AudioUnitInitialize(*voice_unit);

Thank you guys so much for patience

[jvalli]

Hi @hieultse03355:

• are the ways you guys set up AudioUnitSetProperty and stream_format AudioStreamBasicDescription still correct? and have these based on any formula? Yes, we are based on Apple's documentation. The Custom audio driver demo app is pretty much the same we do internally, we have a few minor updates and the implementation of the AudioUnitInitialize retry logic, I did not work on that workaround but as far as I heard about it seems to be working.

• are there documentations about these set up ? There is no specific documentation for AudioUnitSetProperty & AudioStreamBasicDescription from our side, this is part of Apple AudioToolbox framework