Device specific problems and the future

[rpattabi]

Device specific problems (DSP!) are the most nagging for audio developers on android. I would like to understand what Oboe is doing to address this problem and start a discussion.

It is important for Oboe team to understand how we, app developers, come to know about DSP (not Digital Signal Processing, of course). We have limited set of devices with us. Cloud infrastructure like Firebase Test Lab is not much help to us as they don't support audio (most devices simply muted, only screen capture videos and no sound, etc.). So we end up learning about DSPs from the worst source possible, our users. Most users just uninstall the app when they encounter audio problems. Some users are nice enough to give us one star review and leave. Of course, responding to those reviews never helped to get any further details. I hope you can understand the remote chance for ourselves to discover DSPs.

So it is not a surprise that we considered to immediately migrate our apps to Oboe when we found out about Quirks Manager. QuirksManager keeps growing and new issues keep showing up. This is good and bad. Good thing is that now we have a single source to workaround these problems. But not all DSPs can be addressed like this.

Basic questions: (1) Why are there DSPs? (2) How to identify existing DSPs? (3) How to prevent DSPs from occurring?

Pretty sure, Oboe team has a good idea about this and working hard on it. In my limited understanding one of the issues may be: Device manufacturers don't or don't have to adhere to Android Compatibility Definition(CDD) fully (e.g. #911).

I believe Android Audio and/or Oboe team can do the following (It would be great to know if they are already working on these things):

1. Create automated audio test suite for the device and chipset manufacturers to easily validate and adhere to CDD. This way, at least the future devices will have more uniform behavior for audio.
2. If there is any unavoidable deviation from CDD, manufacturers should be given a standard way to provide info about this deviation. This could help oboe to provide workarounds more easily.
3. Proactively and continuously collect popular devices (and chipsets) and check quirks in the audio. OboeTester helps a lot but it is still manual work. There shall be and automated test suite for common scenarios to make this scalable and continuous.
4. If these test suites are made available to us app developers, we could contribute and even share test results from the devices available to us. This way of crowd sourcing can improve the coverage to even more devices.
5. Communicate continuously with manufacturers regarding quirks identified this way. I understand Oboe team is already doing this. But I think it is only case by case. Test suite can help with standardizing, simplifying, and even avoiding this communication.
6. Work with Firebase Test Lab, for example, to provide facilities to test audio apps. This will really help us to get access to more devices. We made a few requests to Firebase on this but not heard from them.
7. Work with android test team to provide features for testing audio. We have Espresso and UiAutomator, but they are just for checking UI behavior on devices, but we also need a way to check audio behavior of our apps on devices.

[philburk]

@rpattabi - Thanks for this letter. It is very helpful to hear from developers about this issue. I will be sharing this with our management.

I agree with your sentiment and we are trying to fix this problem.

(1) Why are there DSPs?

Primarily from lack of automated test suites and/or not enough manual testing. We do have lots of automated API tests through CTS. But testing audio hardware and device drivers can involve listening to audio, recording sound, plugging in headphones and other tasks that cannot be easily automated. We are working on it. But it is just harder and often requires specialized equipment.

OboeTester is part of that effort. It can be run manually or from an Intent as part of an automated regression test. OboeTester provides lots of manual options, which is handy for debugging. But we need more automated tests for speaker paths, signal integrity, etc.

If these test suites are made available to us app developers, we could contribute and even share test results from the devices available to us.

I agree that would be helpful. We have gotten many useful reports from app developers. That is one reason OboeTester is open source. But app developers can only test phones that have already shipped. We also need to provide more tests to manufacturers so that they can catch bugs before they ship.

Work with Firebase Test Lab, for example, to provide facilities to test audio apps.

I will investigate that possibility.

Work with android test team to provide features for testing audio. We have Espresso and UiAutomator, but they are just for checking UI behavior on devices, but we also need a way to check audio behavior of our apps on devices.

Can you elaborate on that?

[rpattabi]

It is a subject of exploration regarding what android test team can do for testing audio. I think the possibilities are many once we start thinking in this line. The goal is to improve testability of audio on android.

I thought the following mainly for Instrumentation tests:

1. Espresso/UiAutomator can provide a way to access and change audio settings. For example, there are different volume levels for media, alarm, and notification sounds. The app may have functionality to allow the user to manipulate one or more of these levels. Making this available in instrumentation tests will help validate such features.
2. Another example: We add <item name="android:soundEffectsEnabled">false</item> to app theme to prevent tapping sounds so that they don't interfere with the app. With the way we organize our styles and themes, it is not impossible to mess this up. So it would help if it is possible to validate this through instrumentation tests.
3. Android test team could provide us mock audio devices like mock speaker and mock microphone. For example, a test can capture the sound that would go to speaker and validate it. Similarly, test signal through mock microphone could help simulate input. This will facilitate integration tests on actual devices.
4. With support from OS, there could even be a facility to internally capture what is played through the speakers. This way we can have some hope to find device specific glitches through automated tests on many devices (e.g. through firebase test lab). OS already provides several ways to control display (e.g. scaling or disabling animation through developer options) and many other things (e.g. Don't keep activities, Background process limit, etc.). I always wondered why no option is available for audio (to be fair there is "Disable USB audio routing" but you get the point).
5. I learned from Oboe team that AAudio adds the ability to check xruns (if device supports it). If xruns is made available for instrumentation tests (through espresso for example), this will avoid the need to modify our production code just to extract this info for the sake of testing. As it is difficult to run audio flinger dump during the tests, curated info about running audio could be made available for tests. This will greatly improve testability of audio.

[dturner]

Quick update: @philburk is investigating audio testing at scale