Patchfield is an audio infrastructure for Android that provides a simple, callback-driven API for implementing audio modules (such as synthesizers and effects), a graph-based API for connecting audio modules, plus support for inter-app audio routing. It is inspired by JACK, the JACK audio connection kit.
Patchfield consists of a remote or local service that acts as a virtual patchbay that audio apps can connect to, plus a number of sample apps that illustrate how to implement audio modules and how to manipulate the signal processing graph. The implementation resides entirely in userland and works on many stock consumer devices, such as Nexus 7 and 10.
If you have any questions or comments, don't hesitate to get in touch at https://groups.google.com/d/forum/patchfield. If you implement a great project on top of Patchbay, please tell us about it!
Patchfield requires SDK version 14 or later. It pushes the limits of the current Android audio stack, and it works better on some devices than on others. It is also a very young project that has not yet been tested on a wide range of devices. Further testing and evaluation is needed.
Patchfield has been successfully tested on Nexus 10, Nexus 7 (both the original version and the new one), Nexus S, and HTC-1. On Nexus 4, Patchfield works well as long as the Patchfield service runs with foreground privileges.
Patchfield does not seem to work well on Galaxy Nexus (audio glitches when switching between activities), and Galaxy S4 (frequent glitches). The glitches on Galaxy Nexus seem to be unrelated to Patchfield; problems of this kind have also been reported with other audio software.
Make sure that you have Android NDK version r8e or later, and that
ndk-build is on your search path, then clone the project and build the
native components like this:
git clone --recursive https://github.com/google/patchfield.git
cd patchfield
make(Don't worry if javah reports errors on this initial build.) Now you can
import all projects in the patchfield directory into your development
environment of choice. If you build any native binaries after importing the
projects into Eclipse, make sure to refresh your Eclipse workspace or else
Eclipse may not see the latest version.
Patchfield Library project, the core of Patchfield; contains the
PatchfieldService class and all APIs, as well as utilities.PatchfieldControl Sample control app and remote service configuration; you
will need to install this app if you want to run Patchfield as a remote
service. (Note, however, that the Patchfield service does not depend on this
control app; they are merely being packaged together for convenience.)LowpassSample Sample app illustrating how to implement an audio module
for Patchfield, complete with build files and lock-free concurrency.PcmSample Sample app playing a wav file through Patchfield. Crude but
useful for prototyping and demos.PatchfieldPd Library project providing an audio module implementation
that uses libpd for synthesis. This project is a not a toy; libpd is a
heavy-duty signal processing library, and this project fully integrates it into
Patchfield.PdSample Sample app using PatchfieldPd.Further sample projects are available will be posted soon.
Patchfield consists of a number of components. At its core is a Java class,
Patchfield.java, that manages the audio graph and handles audio input and
output with OpenSL ES. Patchfield.java is meant to be accessed through
PatchfieldService.java, which can run as a remote or local service, i.e.,
you can use Patchfield to route audio signals between different apps, or you
can build apps that just use Patchfield internally.
There are two APIs for interacting with the Patchfield service, a Java client API for querying and manipulating the state of the signal processing graph, and a hybrid C/Java module API for implementing new audio modules such as synthesizers and effects.
Patchfield introduces a new permission, USE_PATCHFIELD. Apps will only be
allowed to bind to PatchfieldService as a remote service if they have this
permission.
For the time being, the Patchfield service will not be available from Google Play yet. The reason is that the API and protocol may change a few more times before they are finalized, and so a published service would come with the risk of drifting out of sync with client apps.
If you want to publish Patchfield-based apps on Google Play, you currently have two options. If your app is self-contained, you can just package a copy of the Patchfield service with your app. Or, if you want to support multiple apps that exchange audio via Patchfield, you can publish your own copy of the Patchfield service on Google Play and document the version of Patchbay that client apps will have to use.
Right now the recommended way of using Patchfield is as a local service packaged with your app. When the API and protocol have stabilized, the service will be published on Google Play, and then inter-app audio routing via the common remote service will be preferable.
The Java client API is defined by two interfaces, IPatchfieldService.aidl
and IPatchfieldClient.aidl. In general, clients do not synthesize or
process audio but merely operate on the signal processing graph. The Patchfield
service implements the IPatchfieldService interface. This interface
provides a collection of functions for querying and manipulating the signal
processing graph. Most clients will implement the IPatchfieldClient
interface, which provides a collection of callbacks that the service will
invoke to inform clients of changes in the graph.
The sample app in PatchfieldControl is a typical example of a client app
that does no audio processing of its own. Rather, it presents a simple user
interface that visually represents the graph and lets users connect or
disconnect audio modules. Through the IPatchfieldClient interface, it keeps
its representation of the graph up to date.
On the Java side, the audio module API consists of one abstract base class,
AudioModule.java. Subclasses of AudioModule must implement a method
that registers the module's audio processing callback with the Patchfield
service, plus another method that releases the native resources held by the
module when they are no longer needed.
The audio processing callback will generally be implemented in C or C++, using
a small C API, defined in audio_module.h, that only has two elements: A
function prototype that the signal processing callback must conform to, and a
function that registers the signal processing callback with the Patchfield
service.
Apps that use audio modules also need to bind to a PatchfieldService
instance because some of the audio module setup requires the
IPatchfieldService interface.
The PatchfieldLowpassSample project contains a typical audio module
implementation as well as a simple app that shows how to use an audio module.
It also illustrates how to set up the build system and how to interact with the
audio processing thread in a thread-safe yet lock-free manner.
Patchfield uses 32-bit float samples; buffers are non-interleaved. The Patchfield service operates at the native sample rate and buffer size of the device. This means that audio modules must operate at the native sample rate and buffer size as well. Native sample rates of 44100Hz and 48000Hz are common, and so audio modules must support both. Moreover, audio modules must be prepared to work with arbitrary buffer sizes.
In particular, apps cannot assume that the buffer size is a power of two. Multiples of three, such as 144, 192, and 384, have been seen in the wild. The Patchfield repository includes a utility library that performs buffer size adaptation for audio modules that use synthesis techniques or audio libraries that cannot operate at the native buffer size.
In order to navigate from the control app to an audio module, long-click on the
icon of the module in the control UI. In order to navigate from an audio module
to the control app, long-click on the back button. An abstract utility class,
PatchfieldActivity.java, implements this behavior and manages the
connection to the Patchfield service.
Apps that use Patchfield should launch with launch mode singleTask. This
deviates from the usual recommendations for Android development, but it is
necessary for the navigation model of Patchfield.
Patchfield incurs no latency on top of the systemic latency of the Android audio stack; the audio processing callbacks of all active modules are invoked in one buffer queue callback of OpenSL ES.
Patchfield started as a 20% project and Google owns the copyright. Still, this project is not an official Google product.