Need x86/mips/armeabi/armv8 binaries for Android

[karthikrg]

Only armv7-eabi is present in the published artifact. Also, I can't see any easy way to build it myself.

[krisskross]

There are instructions on how to build Android from source here. If you need to build a different binary you must modify the agcc file before building.

[krisskross]

To be honest, i'm not sure what the most common platform is. We could have multiple platform releases if that make sense.

What's your experience and thoughts? Should we start building for another platform?

[karthikrg]

I can't figure out how to modify the agcc file to build for other archs. Can you help me here?

Normally Android NDK projects are built by specifying archs in Android.mk. Android supports the following architectures:

arm64-v8a armeabi-v7a armeabi mips mips64 x86 x86_64

The app I'm planning to integrate this library in has users with devices on all these architectures.

[krisskross]

I noticed that a new flag is needed for building LMDB for android, -DMDB_USE_ROBUST=0.

Here's a x86_64 configuration.

$NDK/toolchains/x86_64-4.9/prebuilt/linux-x86_64/bin/x86_64-linux-android-gcc --sysroot=$NDK/platforms/android-21/arch-x86_64 -DMDB_DSYNC=O_SYNC -DMDB_USE_ROBUST=0 $@

Here's a ARM configuration.

$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc --sysroot=$NDK/platforms/android-21/arch-arm -DMDB_DSYNC=O_SYNC -DMDB_USE_ROBUST=0 $@

I was expecting this MIPS configuration to work, but there are compilation errors in hawtjni that i'm not sure how to fix.

$NDK/toolchains/mips64el-linux-android-4.9/prebuilt/linux-x86_64/bin/mips64el-linux-android-gcc --sysroot=$NDK/platforms/android-21/arch-mips64 -DMDB_DSYNC=O_SYNC -DMDB_USE_ROBUST=0 $@

[karthikrg]

This is awesome. Let me try and see if i can fix the mips issues. Thanks!

On Monday, January 4, 2016, Kristoffer Sjögren notifications@github.com wrote:

I noticed that a new flag is needed for building LMDB for android, -DMDB_USE_ROBUST=0.

Here's a x86_64 configuration.

$NDK/toolchains/x86_64-4.9/prebuilt/linux-x86_64/bin/x86_64-linux-android-gcc --sysroot=$NDK/platforms/android-21/arch-x86_64 -DMDB_DSYNC=O_SYNC -DMDB_USE_ROBUST=0 $@

Here's a ARM configuration.

$NDK/toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-gcc --sysroot=$NDK/platforms/android-21/arch-arm -DMDB_DSYNC=O_SYNC -DMDB_USE_ROBUST=0 $@

I was expecting this MIPS configuration to work, but there are compilation errors in hawtjni that i'm not sure how to fix.

$NDK/toolchains/mips64el-linux-android-4.9/prebuilt/linux-x86_64/bin/mips64el-linux-android-gcc --sysroot=$NDK/platforms/android-21/arch-mips64 -DMDB_DSYNC=O_SYNC -DMDB_USE_ROBUST=0 $@

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/54#issuecomment-168615447.

[krisskross]

Did you find the cause of the MIPS compilation errors?

[karthikrg]

Didn't even get so far. Configure fails for me:

[INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ lmdbjni-android --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/src/main/resources [INFO] [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ lmdbjni-android --- [INFO] No sources to compile [INFO] [INFO] --- maven-hawtjni-plugin:1.11:build (default) @ lmdbjni-android --- [INFO] Extracting /Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni/target/lmdbjni-0.4.6-SNAPSHOT-native-src.zip to /Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/target/native-build-extracted [INFO] executing: /bin/sh -c ./configure --disable-ccache --prefix=/Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/target/native-build/target --with-lmdb=/Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/../lmdbjni/src/main/native-package/src CC=/Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/../lmdbjni/src/main/native-package/agcc --host=x86_64-unknown-linux-gnu [INFO] configure: WARNING: unrecognized options: --disable-ccache [INFO] checking build system type... i386-apple-darwin14.5.0 [INFO] checking host system type... x86_64-unknown-linux-gnu [INFO] checking target system type... x86_64-unknown-linux-gnu [INFO] checking for x86_64-unknown-linux-gnu-gcc... /Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/../lmdbjni/src/main/native-package/agcc [INFO] checking whether the C compiler works... no [INFO] configure: error: in /Users/kramgopa/projects/lmdbjni/lmdbjni/lmdbjni-android/target/native-build': [INFO] configure: error: C compiler cannot create executables [INFO] Seeconfig.log' for more details [INFO] rc: 77 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] lmdbjni-project .................................... SUCCESS [ 3.535 s] [INFO] lmdbjni ............................................ SUCCESS [ 9.611 s] [INFO] lmdbjni-android .................................... FAILURE [ 1.807 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 15.062 s [INFO] Finished at: 2016-01-04T15:19:35-08:00 [INFO] Final Memory: 30M/311M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.fusesource.hawtjni:maven-hawtjni-plugin:1.11:build (default) on project lmdbjni-android: build failed: org.apache.maven.plugin.MojoExecutionException: ./configure failed with exit code: 77 -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :lmdbjni-android

[karthikrg]

I think the reason is that android defines cacheflush differently than standard linux for MIPS. mdb.c uses this.

https://android.googlesource.com/platform/bionic/+/1e980b6/libc/arch-mips/bionic/cacheflush.c

I think redefining cacheflush to use the GNU __clear_cache should do the trick.

define CACHEFLUSH(start, length, flags) __clear_cache(start, (char *) (start + length))

[karthikrg]

Also FWIW, I hated hawtjni so I pulled out the code, edited it a bit and made it a regular Android NDK project. I'll post the source once its tested and ready on github.

[krisskross]

Yes, I also ran into the clear_cache issue.

Glad to hear you made it work. Interesting to see the final result when you're done. If possible, maybe we can include the changes into lmdbjni?

[krisskross]

I was just curious if you managed to get this working?

[karthikrg]

Yep. I did. It's involved significant code changes though. I've also added code to pool buffers in java and avoid byte array garbage; and a zero copy dma buffer implementation that works on Android.

I'm running it through my company's open source review team before getting it out there. Due to the nature of these processes may take about a month or so to get it out there.

Thanks, Karthik

On Tuesday, February 2, 2016, Kristoffer Sjögren notifications@github.com wrote:

I was just curious if you managed to get this working?

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/54#issuecomment-178694989.

[krisskross]

Cool, I had some problems earlier to get direct byte buffers working on android due to (I think) memory alignment #21. Would be interesting to hear how you solved that.

I'll keep this issue open until later.

[karthikrg]

Yeah unsafe doesn't work on Android. I implemented it by providing a java.nio.Buffer implementation backed by a JNI allocated byte array.

[phraktle]

Using ByteBuffers could also resolve my other request, #42.

[krisskross]

Did you use NewByteArray or NewDirectByteBuffer? As far as I understand, byte arrays are allocated on the heap?

[karthikrg]

My buffer implementation uses malloc directly. Direct byte buffer alloc is very (50-100x compared to new byte[]) slow in Android, and dalvik/art still track their allocations via a associated pointer.

On Wednesday, February 3, 2016, Kristoffer Sjögren notifications@github.com wrote:

Did you use NewByteArray or NewDirectByteBuffer? As far as I understand, byte arrays are allocated on the heap?

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/54#issuecomment-179240917.

[kittinunf]

Do you guys have any trick to make compilation successful on mips?

../../../vendors/lmdb/libraries/liblmdb/mdb.c:76:12: error: conflicting types for 'cacheflush'
extern int cacheflush(char *addr, int nbytes, int cache);
           ^
/Users/kittinunf/Library/Android/sdk/ndk-bundle/platforms/android-16/arch-mips/usr/include/unistd.h:183:12: note: previous declaration is here
extern int cacheflush(long start, long end, long flags);

[karthikrg]

Make it a macro as I described above and use it!

[kittinunf]

@karthikrg Sorry I must have missed that last line. Yeah, thanks it works!