how to package .mhr file into android apk?

[wangii]

it seems the library searches .mhr files on file system. so where should I put them into the android package?

[kcat]

I'm not quite sure how Android packages are utilized. By default, the Android (non-Windows) version of OpenAL Soft will look in /usr/share/openal/hrtf, /usr/local/share/openal/hrtf, $HOME/.local/share/openal/hrtf, and the current dir. I don't know if there's a way to package APKs to have files put in those locations when extracted (are they extracted? when installed or executed?), or behave as if files were there (i.e. virtual directories). Do you have any insight about that?

[wangii]

unless rooted, these paths are not available. the normal way is to put files under assets folder of the app, and use AAssetManager ndk ref to open/read files. the annoying part is that the lib users would have to pass jvm for AAssetManager_fromJava for it to work.

I think an easier way is to use objcopy to turn files into obj files (.o), then embed into the shared lib.

how do you think?

[fatalfeel]

I do this for you ~~~download android version, it also have mp3 java decoder example http://fatalfeel.blogspot.tw/2013/09/openal-hrtf-3d-sound.html

[wangii]

@fatalfeel thanks! I came across your page last night, thought it was outdated :) It might not work for me out of box, since I operate openAL not in the main thread. Will get back to you when I have some result.

[kcat]

I do this for you ~~~download android version

OpenAL Soft already builds on Android. The problem here is the inability to read the mhr files (or any files at all, apparently) that are packaged in the APK from native C code. Everything I'm finding is saying it requires some "Java side" code to interact with (which requires a JNIEnv* that an app would need to pass in, since shared libs don't implicitly get one) and do some ugly things to get an fd that you can get a FILE* from, then messing about with file offsets and junk and making sure the resource is uncompressed. Really makes me wonder why there aren't any hooks in their libc stdio implementation to access APK resources as read-only files in a virtual directory (could even transparently handle compressed resources).

I think an easier way is to use objcopy to append the file into the shared lib's data section. how do you think?

Earlier versions did compile the default HRTF datasets into the lib itself, although it's not an insignificant amount of space (about 100KB for both, where the release-build lib is currently about 700KB; so combined, those datasets would make up a full 1/8th of the lib size which will be there in memory even if never used). There are other datasets that aren't so gratuitous, for instance the IRCAM sets create 14KB mhr files, so about 28KB for both which is much more reasonable. Though that comes at the cost of having fewer measurements, for slightly less accurate positioning.

[fatalfeel]

I load mhr use jni C code not java~~~~ java only pass the apk file name

[kcat]

I load mhr use jni C code not java~~~~ java only pass the apk file name

Can you provide patches for your changes? .diff or .patch files, if possible. The snippets you show are difficult to follow and don't make a whole lot of sense. Section 3 hard-codes a path that is very unlikely to work on other peoples' systems (and breaks XDG-based systems), section 4 will create duplicate entries and entries for files it fails to load, both of which causing a crash when it tries to use them, and everything under the ///Android system/// part is hard-coding build-time macros and file lists that may not be valid for different compilers, and will not be correct when OpenAL Soft updates (either adding or removing sources, or changing the version number). I'm also not seeing where it handles reading mhr files from the APK.

[fatalfeel]

apk is zip file, unzip it to memory, and load it~~~~ Get_unz this function

[wangii]

can we have an option if to include the .mhr files into libs? i think for current phones, few hundreds k of memory not a big deal. besides, if developers want to enable hrtf, they'll have to load into memory anyway.

[fatalfeel]

sure, now its ur show time, wait ur code~~~~

[wangii]

I prefer to be approved first.

I propose to use shm_open to create fds, then mmap the addresses. they can be used by LoadHrtf00 and LoadHrtf01. the solution is simple but not sure if works for windows.

I'll need some help with CMake file if the idea is ok.

[kcat]

I'm looking into making a portable build option to embed the default mhr files into the lib. I've got the embedding itself working more or less, on both Windows and Linux (which should cover Android too), I just need to add code to read and include them in the HRTF lists.

[fatalfeel]

mhr outside is better then we can add it ourself

[wangii]

@kcat looking forward your implementation! just curious, what's the exact technique you are using?

@fatalfeel I think Kcat is going to have some preprocess scripts that can be easily applied to your own .mhr files, and to include .mhr into lib, it's an -DINCLUDE_MHR kind of option. you can turn it off when invoking cmake.

[kcat]

just curious, what's the exact technique you are using?

On Windows, I'm using an RC file to include the mhr files as a resource in the DLL, then using FindResource, LoadResource, etc to get it as a chunk of memory. On other systems, I'm using ld -r -b binary ... to create object files with the mhr file data inside, and which defines the symbols _binary_..._start and _binary_..._end at the start and end of the binary data (I also use objcopy --rename-section ... to change it to a read-only data section, which should make it use cached file pages at runtime instead of taking up actual heap memory).

Currently it just embeds the hrtf/default-44100.mhr and hrtf/default-48000.mhr files. You could replace these at build time if you wanted to include a different set.

[wangii]

great! Can't wait to test & view your codes.

[kcat]

Code for it is pushed now. Specify -DALSOFT_EMBED_HRTF_DATA=TRUE to cmake, and it'll embed default-44100.mhr and default-48000.mhr into the lib and include them with HRTF enumeration (as Built-In * entries).

[wangii]

bad news, not working smoothly out of box. the playing result is like this: 20160713_003.amr.zip

without enabling HRTF, it plays well.

additional info:

• compiled with toolchain file: https://github.com/taka-no-me/android-cmake/blob/master/android.toolchain.cmake
• cmake command line: cmake .. -DCMAKE_TOOLCHAIN_FILE=../cmake/android.toolchain.cmake -DANDROID_NDK=~/Library/Android/sdk/ndk-bundle -DCMAKE_BUILD_TYPE=Release -DANDROID_ABI=armeabi -DANDROID_NATIVE_API_LEVEL=android-18 -DANDROID_STL=gnustl_static -DALSOFT_EMBED_HRTF_DATA=TRUE
• Tested on devices: Kindle File 5g, Huawei C8817.
• voice format: 16b@22050, 2ch, format mp3. I used libmpg123 to do decoding.

(btw, if emit stereo data to source, 3d doesn't work. I read somewhere said otherwise, not a big problem though)

[kcat]

Hmm, sounds like the audio is dropping out. I'm not very familiar with the CPU power of Android devices, so I can't tell if maybe they're just not strong enough to handle HRTF mixing. Can you set the ALSOFT_LOGLEVEL env var to 3 when running the app, and provide the output from stderr?

[kcat]

And yes, stereo (non-mono) sounds won't move when setting AL_POSITION and such. The buffer channels are virtualized though, so it'll sound like they're playing from speakers in front of the listener. The effect is more noticeable with surround sound formats like AL_FORMAT_QUAD and AL_FORMAT_51CHN, as the rear channels will sound as if they're behind the listener and give the appropriate surround effect.

For stereo formats in particular, there is the AL_EXT_STEREO_ANGLES extension, which allows you to "pan" or rotate the stereo channels toward a particular (horizontal) direction.

[wangii]

I guess it's not a CPU problem. To get the log printed out, I've done:

• Set LogLevel unconditionally to LogTrace in ALc.c.
• Removed getenv section in alc_initconfig, ALc.c
• in alMain.c, defined AL_PRINT to use android log function: #define AL_PRINT(T, MSG, ...) __android_log_print(4, "ALLIB", "AL lib: %s %s: "MSG, T, __FUNCTION__ , ## __VA_ARGS__)
• updated cmake file to link against -landroid -llog

Anything I've done wrong above?

I/ALLIB   ( 7708): AL lib: (II) alc_initconfig: Supported backends: opensl, null, wave
I/ALLIB   ( 7708): AL lib: (II) ReadALConfig: Loading config /etc/openal/alsoft.conf...
I/ALLIB   ( 7708): AL lib: (II) ReadALConfig: Loading config /etc/xdg/alsoft.conf...
I/ALLIB   ( 7708): AL lib: (II) GetProcPath: Got: /system/bin
I/ALLIB   ( 7708): AL lib: (II) ReadALConfig: Loading config /system/bin/alsoft.conf...
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key disable-cpu-exts not found
I/ALLIB   ( 7708): AL lib: (II) FillCPUCaps: Extensions: -none-
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key rt-prio not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key resampler not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key trap-al-error not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key trap-alc-error not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key reverb/boost not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key reverb/emulate-eax not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key drivers not found
I/ALLIB   ( 7708): AL lib: (II) alc_initconfig: Initialized backend "opensl"
I/ALLIB   ( 7708): AL lib: (II) alc_initconfig: Added "opensl" for playback
I/ALLIB   ( 7708): AL lib: (II) alc_initconfig: Initialized backend "null"
I/ALLIB   ( 7708): AL lib: (II) alc_initconfig: Initialized backend "wave"
I/ALLIB   ( 7708): AL lib: (WW) alc_initconfig: No capture backend available!
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key excludefx not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key default-reverb not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key channels not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key sample-type not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key frequency not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key periods not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key period_size not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key sources not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key slots not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key sends not found
I/ALLIB   ( 7708): AL lib: (II) alcOpenDevice: Created device 0x42b1c020, "OpenSL"
I/ALLIB   ( 7708): AL lib: (II) UpdateDeviceParams: ALC_FREQUENCY = 44100
I/ALLIB   ( 7708): AL lib: (II) UpdateDeviceParams: ALC_HRTF_SOFT = 1
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key frequency not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key sends not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key hrtf not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key hrtf-paths not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key hrtf_tables not found
I/ALLIB   ( 7708): AL lib: (II) DirectorySearch: Searching / for *.mhr
I/ALLIB   ( 7708): AL lib: (II) DirectorySearch: Searching /usr/local/share/openal/hrtf for *.mhr
I/ALLIB   ( 7708): AL lib: (II) DirectorySearch: Searching /usr/share/openal/hrtf for *.mhr
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Loading Built-In 44100hz...
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Detected data set format v1
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Loaded HRTF support for format: Stereo 44100hz
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Adding built-in entry "Built-In 44100hz"
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Loading Built-In 48000hz...
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Detected data set format v1
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Loaded HRTF support for format: Stereo 48000hz
I/ALLIB   ( 7708): AL lib: (II) AddBuiltInEntry: Adding built-in entry "Built-In 48000hz"
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key default-hrtf not found
I/ALLIB   ( 7708): AL lib: (II) UpdateDeviceParams: Pre-reset: *Stereo, Float, *44100hz, 1024 update size x4
I/ALLIB   ( 7708): AL lib: (II) UpdateDeviceParams: Post-reset: Stereo, Signed Short, 44100hz, 2048 update size x2
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key stereo-mode not found
I/ALLIB   ( 7708): AL lib: (II) GetConfigValue: Key hrtf-mode not found
I/ALLIB   ( 7708): AL lib: (II) aluInitRenderer: HRTF enabled, "Built-In 44100hz"
I/ALLIB   ( 7708): AL lib: (II) alcCreateContext: Created context 0x42b016a0

[wangii]

don't worry about the stereo format. currently I set the type to AL_FORMAT_MONO16 and doubled the frequency from 22050 to 44100.

[wangii]

It turned out to be a cpu/ABI issue. I recompiled with -DANDROID_ABI=armeabi-v7a, it works well now on Kindle File 5g. on my Huawei C8817D, it's better although not perfect.

[kcat]

Anything I've done wrong above?

Nothing stands out as wrong to me, though I'm not overly familiar with Android's logging functions. Either way, the output itself looks okay. But I do notice it mentions there's no CPU extensions, meaning it's not compiled with Neon support, which can certainly affect performance. Since you say using armeabi-v7a improves it, that makes me wonder if that's enabling Neon support. If that's the case, then it seems those devices are just on the edge of having enough CPU power for HRTF processing.

Given that, there's a few things to note:

• Normally with HRTF enabled, the device sets up a virtual cube channel array for certain things that aren't directly HRTF-capable to mix to (e.g. effects, B-Format buffers). Those virtual channels are then mixed with HRTF to the "real" output. This allows things which normally don't work with HRTF to still play in 3D through it. However this also means there are a minimum of 8 channels being mixed with HRTF at any given time (8 for the cube, plus whatever you play).
• There is a config option (hrtf-mode = basic) to make sources mix to that virtual cube instead of directly with HRTF. This is (should be) a bit more efficient since sources would use a simpler mixing method, leaving only the 8 cube channels to use HRTF. This does however have the side effect of making the perceived positioning a bit less accurate. And if the 8 channels alone are still too much, it's not going to fix it.
• You could comment out the code that mixes the cube to the output, preventing those 8 channels from being mixed with HRTF (make sure to leave the if check so it'll still take the branch, just comment out what's in the branch). This will obviously improve performance since it's 8 less channels using HRTF, but it will cause sounds that rely on the cube to no longer be heard. And if you do play 8 sources or more, you'll end up requiring the same amount of CPU power again.

As an alternative to HRTF, OpenAL Soft also has a stereo-to-binaural filter. This essentially does a normal stereo mix, and then applies crossfeed filters. This doesn't give you "real" 3D (or even surround sound) audio, instead it's just basic stereo, but it does make listening on headphones a bit more pleasant, and is a lot more efficient. This can be enabled with the cf_level config option.

[wangii]

@kcat thank you very much for the great work. I truly appreciate.