Implement audio file decoding support on Android using MediaCodec

[kevinstadler]

To improve audio file format support on most platforms, starting with 0d8287d the SoundFile class uses the JavaSound system to decode audio files, which however does not exist on Android. In order to achieve the same format coverage as on full JavaSE platforms, a decoding routine mimicking the JavaSound one but based on the Android MediaCodec class would have to be implemented.

The basic support for 16 bit wavs and aiffs on Android is not affected by this change, however there is the chance that the SoundFile class in its current state will not even be run because of imports that are missing on Android. I would be super grateful if @codeanticode, @Calsign, @daniel-tran or other users with a running Android setup could take the current development build for a test ride and report back any issues?

(Once some proper unit tests have been added, it would also make sense to add a android-based test target to the Github deploy workflow.)

[daniel-tran]

I tried the development build on my device (Android 13 using Processing 4.2) and the sketch is stuck building indefinitely with the following error in the console:

Exception in thread "Thread-11" java.lang.NullPointerException: Cannot read the array length because "list" is null
    at processing.app.Library.wrapFiles(Library.java:427)
    at processing.app.Library.getApplicationExports(Library.java:437)
    at processing.mode.android.AndroidBuild.copyImportedLibs(AndroidBuild.java:839)
    at processing.mode.android.AndroidBuild.createAppModule(AndroidBuild.java:481)
    at processing.mode.android.AndroidBuild.createProject(AndroidBuild.java:262)
    at processing.mode.android.AndroidBuild.build(AndroidBuild.java:223)
    at processing.mode.android.AndroidMode.handleRunDevice(AndroidMode.java:294)
    at processing.mode.android.AndroidEditor$17.run(AndroidEditor.java:421)

The sample sketch I'm using is the same test code from #70. Perhaps I may have missed something? (I was also having issues trying to update the Android SDK earlier, though I'd like to think that isn't related)

[kevinstadler]

Thanks so much for the swift feedback! Yeah not really sure what to make of this error message, maybe @codeanticode has an idea? Either way I will try running it on a pristine Android mode setup myself when I get around to it, hopefully before the end of the month.

[kevinstadler]

I get the same error on a fresh Android mode install, looks like it is a general problem with libraries that bundle any native libraries: processing/processing-android#738

When removing all native library directories, you get the following error instead:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:checkDebugDuplicateClasses'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.CheckDuplicatesRunnable
   > Duplicate class com.portaudio.BlockingStream found in modules jetified-jportaudio (jportaudio.jar) and jetified-jsyn-17.1.0 (jsyn-17.1.0.jar)
     Duplicate class com.portaudio.DeviceInfo found in modules jetified-jportaudio (jportaudio.jar) and jetified-jsyn-17.1.0 (jsyn-17.1.0.jar)
     Duplicate class com.portaudio.HostApiInfo found in modules jetified-jportaudio (jportaudio.jar) and jetified-jsyn-17.1.0 (jsyn-17.1.0.jar)
     Duplicate class com.portaudio.PortAudio found in modules jetified-jportaudio (jportaudio.jar) and jetified-jsyn-17.1.0 (jsyn-17.1.0.jar)
     Duplicate class com.portaudio.StreamInfo found in modules jetified-jportaudio (jportaudio.jar) and jetified-jsyn-17.1.0 (jsyn-17.1.0.jar)
     Duplicate class com.portaudio.StreamParameters found in modules jetified-jportaudio (jportaudio.jar) and jetified-jsyn-17.1.0 (jsyn-17.1.0.jar)
     Duplicate class javazoom.spi.PropertiesContainer found in modules jetified-mp3spi-1.9.5.4 (mp3spi-1.9.5.4.jar) and jetified-vorbisspi-1.0.3 (vorbisspi-1.0.3.3.jar)

Instead of trying to solve this inside the Android mode build process it might be easier to just create a separate Android release without any of the (anyway mostly unnecessary) bundled jars and native libraries..

[kevinstadler]

@daniel-tran could I ask you to try running any example sketch (ideally one of the Oscillators and, if that works, one of the Soundfile ones) with this sound-android.zip? The sketch should definitely build successfully with this one, I just haven't been able to get any sketch to actually run on my phone with my current Android mode setup..

[daniel-tran]

I've replaced the sound library with the one from the .zip file and the "SineWave" & "SimplePlayback" example sketches build successfully. However, they proceed to crash immediately when opened on my Android device with the following error logged in the console (this one is from the "SimplePlayback" sketch, though the core error is essentially the same for both examples):

FATAL EXCEPTION: Animation Thread
Process: processing.test.simpleplayback, PID: 7168
java.lang.NoClassDefFoundError: Failed resolution of: Ljavax/sound/sampled/DataLine$Info;
    at processing.sound.Engine.probeDeviceOutputLine(Unknown Source:0)
    at processing.sound.Engine.selectOutputDevice(Unknown Source:82)
    at processing.sound.Engine.selectOutputDevice(Unknown Source:45)
    at processing.sound.Engine.<init>(Unknown Source:43)
    at processing.sound.Engine.<init>(Unknown Source:1)
    at processing.sound.Engine.getEngine(Unknown Source:10)
    at processing.sound.Engine.getEngine(Unknown Source:1)
    at processing.sound.SoundObject.<init>(Unknown Source:10)
    at processing.sound.AudioSample.<init>(Unknown Source:0)
    at processing.sound.SoundFile.<init>(Unknown Source:0)
    at processing.sound.SoundFile.<init>(Unknown Source:1)
    at processing.test.simpleplayback.SimplePlayback.setup(SimplePlayback.java:36)
    at processing.core.PApplet.handleDraw(PApplet.java:1878)
    at processing.core.PSurfaceNone.callDraw(PSurfaceNone.java:478)
    at processing.core.PSurfaceNone$AnimationThread.run(PSurfaceNone.java:518)
Caused by: java.lang.ClassNotFoundException: Didn't find class "javax.sound.sampled.DataLine$Info" on path: DexPathList[[zip file "/data/app/~~kt51B7RJwOX07o_LI4n69w==/processing.test.simpleplayback-9KgFPbry35mKQ57batKaxQ==/base.apk"],nativeLibraryDirectories=[/data/app/~~kt51B7RJwOX07o_LI4n69w==/processing.test.simpleplayback-9KgFPbry35mKQ57batKaxQ==/lib/arm64, /system/lib64, /system_ext/lib64]]
    at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:259)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:379)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
    ... 15 more

Running the sketches in Java mode still seems to work.

[kevinstadler]

I've replaced the sound library with the one from the .zip file and the "SineWave" & "SimplePlayback" example sketches build successfully. However, they proceed to crash immediately when opened on my Android device with the following error logged in the console (this one is from the "SimplePlayback" sketch, though the core error is essentially the same for both examples):

Thanks, that's really useful! If this is not getting boring for you yet, here would be an updated sound-android.zip to try out which should make it past that Exception. Hoping I'll manage to get Android mode up and running here to be able to test myself...

[daniel-tran]

The new .zip file works much better now and the "SineWave" & "SimplePlayback" example sketches don't crash anymore.