Android: Revamp ofSoundStream (Independent Buffer Sizes for audio in / out)

[cerupcat]

On Android, often the mic input needs a much larger buffer size than the audio output. It would be nice to be able to set independent sizes for audio in and audio out. Currently, if the mic input is set to low latency (to get lower output latency) the mic is choppy sounding. When set higher then it's ok, but then there's large latency on output.

See related OF thread: http://forum.openframeworks.cc/index.php/topic,12723.new.html#new

[thiagohersan]

also, I don't think it's possible to write an app with just audio input, without output.

I've been meaning to look into that, so I'll take a look at this too.

[cerupcat]

Yeah, I noticed that as well. It'll crash if you try to do 1 audio input and 0 audio output.

We might want to try completely revamping Android audio to use OpenSL instead which is available on Android 2.3+ and is a native audio solution instead of using java. There's a good article here on it: http://audioprograming.wordpress.com/2012/03/03/android-audio-streaming-with-opensl-es-and-the-ndk/, with a sample project here: https://bitbucket.org/victorlazzarini/android-audiotest. CSound and libpd both are using openSL.

[bilderbuchi]

@admsyn could you take a look at this?

[danomatika]

Yes, I second OpenSL and point y'all to Peter Brinkmann's libpd branch that is using it: https://github.com/libpd/libpd/tree/semaphores He suggest his layer as an option for OF.

We all want faster ofxPd+Android: http://createdigitalnoise.com/discussion/3590/android-audio-output-ok-but-audio-input-choppy

[nettoyeurny]

The OpenSL layer of libpd is not specific to libpd at all. Here it is: https://github.com/libpd/libpd/blob/semaphores/jni/opensl_io.h

It basically provides a callback-driven API on top of OpenSL that handles all the complexity of OpenSL under the hood.

The semaphores branch is under active development right now, but it's already more performant and more robust that the version in the stable branch. I expect the dust to settle soon.

[admsyn]

@bilderbuchi yepp, looking into it :)

@cerupcat, @danomatika, OpenSL does look like the sensible way to go. Thanks for the code sample @nettoyeurny!.

@arturoc Thoughts?

[arturoc]

i think we should keep the java version for a couple of versions more to be able to have compatibility with 2.2 which everything else has, and provide openSL for 2.3 and higher

the problem with not being able to use input without output is because of how the circular buffer is used but it should be easy to solve. once this is solved you can have different buffer sizes for in/out by having different ofSoundStream objects.

This can be problematic though, if you want to output the sound from the input since having different threads you'll need to sync the input and output manually (that's what the circular buffer does currently)

I've used this code for in/out without problem but will take a look when i have a moment

[nettoyeurny]

FYI: Part of the point of my OpenSL layer is to take the pain out of the synchronization of input and output. The latest version uses ideas from the recent I/O talk to accomplish that in a way that's robust and tries to reduce latency as much as reasonable.

[arturoc]

thanks! looks super easy to use, it should be pretty direct to implement a new version of the ofxAndroidSoundStream using that layer

[cerupcat]

I've implemented a ofxAndroidSoundStream version using OpenSL and @nettoyeurny's layer. I've only tested with having both in and out, but it's working very well with low latency. I no longer have choppy input audio with high latency. I don't have it on github yet, but you can download it here (it's just the audio files): https://www.dropbox.com/s/vtf2oaizth6r4wc/ofxAndroid_OpenSL.zip

To compile, you also need to add the following to the PLATFORM_LDFLAGS: -llog -lOpenSLES

I removed the java audio thread, but kept in the headphone broadcaster. So, it'll only work with Android 2.3+, but the older code could easily be added back in to work with Android 2.2. This is for another discussion, but we might want to consider making 2.3 the minsdk since Android 2.2 is now < 5% (for our apps only 1%) of the user-base while 2.3 is 30%.

[nettoyeurny]

Glad to hear it's working!

FYI: I think I'll rename the OpenSL module because by now it's so far from Victor Lazzarini's original streaming implementation that it's too confusing to use the same name, and then I'll package it as a separate git repository so it'll be easier to reuse outside of libpd. I'm hoping to take care of that today or tomorrow.

[nettoyeurny]

The renamed and slightly revised library is now available at https://github.com/nettoyeurny/opensl_stream.

[cerupcat]

I've been running into a number of issues with Android audio.

I'd love to use https://github.com/google/oboe as the audio framework since it seems to handle many of the issues I'm seeing on various devices, but it requires a minimum NDK version of 17. OF is currently on 15 and updating to 17 produces many errors.