Make it possible to configure the microphone input stream for lowest latency

[hakoptak]

Describe the project you are working on: A microphone-driven mobile game.

Describe the problem or limitation you are having in your project: My game needs to detect external sound beeps as fast as possible. This requires the lowest possible input latency. The standard Android OpenSL ES configuration does not guarantee the lowest possible input latency due to input filtering.

Describe the enhancement and how it helps to overcome the problem or limitation: To guarantee the lowest possible input latency, the OpenSL ES AudioRecorder must be configured with the SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION flag.

Source: https://developer.android.com/ndk/guides/audio/opensl/opensl-prog-notes

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams: To configure the AudioRecorder one has to add the following lines to the audio_driver_opensl.cpp file:

// Obtain the configuration interface for the recorder
SLAndroidConfigurationItf configItf;
(*recorder)->GetInterface(recorder, SL_IID_ANDROIDCONFIGURATION, &configItf);

// Set the recording flag
SLuint32 recPreset = SL_ANDROID_RECORDING_PRESET_VOICE_RECOGNITION;
(*configItf)->SetConfiguration(configItf, SL_ANDROID_KEY_RECORDING_PRESET, &recPreset, sizeof(SLuint32));

It would ideal to define an "Input Latency" property in the "Project Settings > Audio" section and define some presets for the input latency: LOW, DEFAULT. How this translates to other platforms is up to the experts.

If this enhancement will not be used often, can it be worked around with a few lines of script?: This is low-level OpenSL ES so unfortunately not.

Is there a reason why this should be core and not an add-on in the asset library?: This is a core feature.

[hakoptak]

I should have mentioned that a similar flag is available on iOS: AVAudioSessionModeMeasurement

This flag could be set in app_delegate.mm:

// Put system in measuring mode
NSError *error;
[[AVAudioSession sharedInstance] setMode:AVAudioSessionModeMeasurement error:&error];

"Use this mode for apps that need to minimize the amount of system-supplied signal processing to input and output signals." Source: https://developer.apple.com/documentation/avfoundation/avaudiosessionmodemeasurement?language=objc

So this feature can be implemented as mobile-only.

[m4gr3d]

@hakoptak it would be useful if you could also provide the performance metrics you're trying to achieve. For example what's the latency that you're seeing now and how small are you expecting it to be if this proposals is implemented.

Providing a reference project for measuring the latency would help as well.

[hakoptak]

@m4gr3d: good points.

Performance metrics My game is reaction based. When the player receives a certain visual cue, he/she has to shout 'beep'. This beep must be at least 50 ms long before it can be detected. This time window is the ideal case. To feel natural the visual response must be under 100 ms. So if the extra input latency (due to noise reduction) is below 50 ms then it should be fine. But the lower the better.

Source: https://stackoverflow.com/questions/6880856/whats-the-minimum-lag-detectable-by-a-human

Measured latency To test the effect of the microphone configuration I used the following project:

https://github.com/hakoptak/SuperpoweredLatency

It measures the round trip latency with a beep sound (speaker > microphone). On my Moto G3 I found a latency of 48 ms for the fast path and 63 ms for the slow path, a 15 ms difference. However, I have to mention that the measurements were quite unstable. It could be possible that the fast path switched over to the slow path during measurement, and vice verse. Not sure why this happens.

Why this setting?

For my device the latency is okay, but there are many Android devices out there with far worse metrics (see: https://superpowered.com/latency). For these devices 15 ms or more could mean a lot for the gameplay.