search

google / oboe

Oboe is a C++ library that makes it easy to build high-performance audio apps on Android.
Apache License 2.0
3.72k stars 571 forks source link

Input gives silence on AAudio except for VoiceCommunication preset (OpenSL input is fine for any input preset) #1110

Closed rpattabi closed 3 years ago

rpattabi commented 3 years ago 
Android version: 10 (API 29)
Android device : Samsung S9+ / samsung SM-G965F (samsung star2lte)
         Build : QP1A.190711.020.G965FXXSBETH2
   Fingerprint : samsung/star2ltexx/star2lte:10/QP1A.190711.020/G965FXXSBETH2:user/release-keys
Oboe version   : At commit ee39453cb1637460f7c4321f9d4d7c4f63d544c0

App name used for testing: A custom app which has foreground UI which runs automated tests which exercise different audio apis and input presets.

Short Description

On the mentioned device, we find that AAudio streams for any input preset (except VoiceCommunication) received digital silence while OpenSL streams worked fine.

Verifying Silence in the Input

We wrote a set of tests to capture input and determine if input is actually working or just received digital silence. This is checked by calculating the RMS of all the samples in callbacks. On real devices, irrespective of input preset or effects processing, there should always be some signal as indicated by RMS. If RMS of data in input callback is exactly zero, it implies digital silence. This could only be caused by the system silencing the input.

Results

We ran these tests on both AAudio and OpenSL. The following are the results for the mentioned device:

AudioApi   InputPreset               Input Behavior
----------------------------------------------------
AAudio     VoiceCommunication        Works
AAudio     Generic                   Digital Silence
AAudio     VoiceRecognition          Digital Silence
AAudio     Unprocessed               Digital Silence
AAudio     Camcorder                 Digital Silence
AAudio     VoicePerformance          Digital Silence

OpenSL     VoiceCommunication        Works
OpenSL     Generic                   Works
OpenSL     VoiceRecognition          Works
OpenSL     Unprocessed               Works
OpenSL     Camcorder                 Works

Full log of the test run is here. If you look for NativeTest tag, you'll find the failed and succeeded tests. Failed tests log the RMS for each callback. Here is an entry of a failed test. As you see, for AAudio + VoiceRecognition stream received digital silence for over 3 seconds.

2020-11-24 08:37:26.000 25785 D/NativeTest: Expected: (nonSilentBins) >= (percentage(rmses.size(), MIN_DESIRED_AUDIBLE_BIN_PERCENTAGE)), actual: 0 vs 376
2020-11-24 08:37:26.000 25785 D/NativeTest: rms = 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
2020-11-24 08:37:26.000 25785 D/NativeTest: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
2020-11-24 08:37:26.000 25785 D/NativeTest: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
2020-11-24 08:37:26.000 25785 D/NativeTest: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
2020-11-24 08:37:26.000 25785 D/NativeTest: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
2020-11-24 08:37:26.000 25785 D/NativeTest: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest:  0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2020-11-24 08:37:26.000 25785 D/NativeTest: [  FAILED  ] AudioApis_InputPresets/InputStreamTest.inputShouldNotBeSilence/AAudio_VoiceRecognition, where GetParam() = (4-byte
2020-11-24 08:37:26.000 25785 D/NativeTest:  object <02-00 00-00>, 6) (3618 ms)

We also log audio stream lifecycle and parameters using OboeStream tag. In this tag, you'll find the average RMS for each second of input audio. The following shows OpenSL stream with VoiceRecognition preset. Notice averageRMSes are non-zero.

2020-11-24 08:37:05.000 25785 D/OboeStream: INPUT state=Started preset=VoiceRecognition api=OpenSL channels=1 format=Float samplerate=48000 convQlty=Medium perfMode=LowLatency xrunsSupport=false deviceId=8 addr=0x73246926e0
2020-11-24 08:37:05.000 25785 D/OboeStream: buffer=2048 capacity=2048 burst=1024 callback=1024 frames
2020-11-24 08:37:05.000 25785 W/OboeStream: InputStream: initial callback latency (#0): 67 ms
2020-11-24 08:37:06.000 25785 D/OboeStream: InputStream: addr=0x73246926e0 callbacks=#0..47 (47) averageRMS=0.00962057
2020-11-24 08:37:07.000 25785 D/OboeStream: InputStream: addr=0x73246926e0 callbacks=#47..94 (47) averageRMS=0.01382109

Additional Context

This device is from one of our users. Google Assistant is off (we verified it).

As per Sharing Input, an app may receive based on various conditions. But our test app has activity visible during the test run and we expect it to receive audio input for any audio api and input preset. We don't find any reason why the app should receive silence as per any of the clauses mentioned in this document.

VoiceCommunication is privacy sensitive and it received signal through AAudio. If other presets receive silence due to any restriction, it is not clear why we get signal through OpenSL for any preset.

Questions

We are wondering what can be done from our end to address this issue in our production app. One obvious workaround is to switch to OpenSL. As we can't know which devices may have this problem, we would be forced to switch to OpenSL for all devices (we suspect that this would bring in other problems).

philburk commented 3 years ago

Thank you for this report. I wonder if it is specific to MMAP.

Can you please do a quick check with OboeTester? Use TEST INPUT or "RECORD and PLAY". Uncheck MMAP box. Select "InPreset:" Then start recording. Do you get a signal?

It is possible that some app may be running in the background, or has left the system in an odd state that has suppressed the input. If you reboot does it change the test result?

rpattabi commented 3 years ago

It took time to get the results from the user. Here they are:

I wonder if it is specific to MMAP.

Spot on! We updated our tests to check input streams with and without MMap. Our user's results show that MMap tests fail on this device. The similar tests without MMap work on this device. We also ran our tests on other devices which support MMap. They succeed irrespective of MMap. Also we don't face this problem with OpenSL which don't have MMap support. So this issue is specific to MMap on this device.

To workaround this problem, we can avoid MMap as we think MMap is mainly to reduce latency and for our app lowest latency is not critical need. We hope this won't create other problems. (To disable MMap, we'll use AudioExtensions from OboeTester code. Please consider exposing this through oboe)

Or is it the one for QuirksManager?

Can you please do a quick check with OboeTester?

We don't have direct access to this device. We think it will be difficult to ask our user to try OboeTester. We give him our test apk which has just one button that runs our tests. These tests automatically check the input as mentioned earlier and accordingly does remote logging of success or failure (with user's consent). Most users who report the problem don't have time or patience even for this.

(Access to devices is probably the biggest problem for us indie audio developers. Even firebase test lab and amazon device farm don't help much as they have very few device models. Even when they have the model, they don't have required version. As per sharing input, audio behavior differs wildly across android versions. Even then, the device clouds aren't designed with audio apps in mind. So android audio team's every bit of effort to standardize the audio behavior and stricter enforcement across device manufacturers is critical for us. Please excuse this digression.)

If you reboot does it change the test result?

Our user kindly tried this and informed us that rebooting didn't help. Though problematic apps, if any, may restart their services upon reboot, as per the above test results we think this problem is due to MMap implementation of this device instead of input sharing.

dturner commented 3 years ago

@rpattabi Thank you for supplying such excellent bug reports - would it be possible for you to send me an email to donturner@google.com? I might have something for you.

philburk commented 3 years ago

Thanks for the report. We should update the QuirksManager for this device/version. We should either disable MMAP INPUT, or coerce the InputPreset to VoiceCommunication. Given that there was also a mono/stereo problem on S9 reported in #824 we should probably just disable MMAP INPUT for S9 on Q. That bug says the S9 has "ro.hardware.chipname=exynos9810".

dturner commented 3 years ago

Decision: disable MMAP input on the S9

dturner commented 3 years ago

@philburk I actually have an S9 so I can add the quirk and test it.

dturner commented 3 years ago

Sorry for the delay in responding on this @rpattabi.

I attempted to reproduce this issue with the following Samsung Galaxy S9 device:

ro.product.brand = samsung
ro.product.manufacturer = samsung
ro.product.model = SM-G965F
ro.product.device = star2lte
ro.product.cpu.abi = arm64-v8a
ro.build.description = star2ltexx-user 10 QP1A.190711.020 G965FXXUFFUE1 release-keys
ro.hardware = samsungexynos9810
ro.hardware.chipname = exynos9810
ro.arch = exynos9810
| grep aaudio = [aaudio.hw_burst_min_usec]: [2000]
[aaudio.mmap_exclusive_policy]: [2]
[aaudio.mmap_policy]: [2]

Reproduction steps:

Results

Preset MMAP Notes
Generic No
Camcorder No
VoiceRecognition Yes
VoiceCommunication No
Unprocessed Yes audio quiet
Performance Yes audio quiet

In all cases I could hear my own voice, albeit quietly on 2 of the presets.

If you note the build number between your test device and mine:

G965FXXSBETH2 - yours G965FXXUFFUE1 - mine

They are different, with mine being later. This leads me to conclude that Samsung have fixed this issue in a later build.

We could add a quirk for the G965FXXSBETH2 and earlier builds. I'll investigate how much work this is (probably not much).