Missing Android N compatibility because of behavior changes

[TheNephilim88]

Hello,

Google changed some things regarding linking/loading libs: https://developer.android.com/preview/behavior-changes.html#ndk

This also affects SQLCipher. E.g. the Test-Suite failed 10 out of 32 tests.

Log show up following:

03-10 09:09:17.497 26856-26869/net.zetetic:provider W/linker: library "libutils.so" ("/system/lib/libutils.so") needed or dlopened by "/data/app/net.zetetic-1/lib/arm/libsqlcipher_android.so" is not accessible for the namespace "classloader-namespace" - the access is temporarily granted as a workaround for http://b/26394120
03-10 09:09:17.497 26856-26869/net.zetetic:provider W/linker: library "libcutils.so" ("/system/lib/libcutils.so") needed or dlopened by "/data/app/net.zetetic-1/lib/arm/libsqlcipher_android.so" is not accessible for the namespace "classloader-namespace" - the access is temporarily granted as a workaround for http://b/26394120
03-10 09:09:17.517 26856-26869/net.zetetic:provider W/linker: library "libnativehelper.so" ("/system/lib/libnativehelper.so") needed or dlopened by "/data/app/net.zetetic-1/lib/arm/libdatabase_sqlcipher.so" is not accessible for the namespace "classloader-namespace" - the access is temporarily granted as a workaround for http://b/26394120
03-10 09:09:17.518 26856-26869/net.zetetic:provider W/linker: library "libandroid_runtime.so" ("/system/lib/libandroid_runtime.so") needed or dlopened by "/data/app/net.zetetic-1/lib/arm/libdatabase_sqlcipher.so" is not accessible for the namespace "classloader-namespace" - the access is temporarily granted as a workaround for http://b/26394120
03-10 09:09:17.518 26856-26869/net.zetetic:provider W/linker: library "libbinder.so" ("/system/lib/libbinder.so") needed or dlopened by "/data/app/net.zetetic-1/lib/arm/libdatabase_sqlcipher.so" is not accessible for the namespace "classloader-namespace" - the access is temporarily granted as a workaround for http://b/26394120

[developernotes]

Hello @TheNephilim88

Thank you for your early report of SQLCipher for Android on Android N. We hope to begin looking into the Android N build soon. Take care!

[splendidbits]

As per https://developer.android.com/preview/behavior-changes.html#ndk

Use of SSL_ctrl symbol from libcrypto.so should be replaced with an app local version. For example, you should statically link libcyrpto.a in your .so file or include your own dynamically libcrypto.so from BoringSSL or OpenSSL in your app.

So instead of OpenSSL being compiled for each arch and being statically linked in the .so, I believe now that the bordingSSL library so (or a?) should be linked directly through the ndk from the user's OS. OpenSSL has been deprecated and removed in favour of boringssl

[developernotes]

Hi @staticfish

We have historically built a static OpenSSL because of the differing versions of OpenSSL bundled across the various Android OS's. We will consider this change in the future.

[TheNephilim88]

Did anyone found time to look into this? I know Android N is still dev preview, but its annoying nonetheless, especially if users dont understand and are giving bad ratings because of it :(

[sjlombardo]

The changes required to address this problem are not trivial and will likely require some major modifications to the library. We are currently in the middle of release preparation for an unrelated update. Once that is complete we will switch over to work on this issue.

I understand that this problem is an inconvenience, however, as you mentioned N is still an early developer release.

[neutron666]

But that's what dev previews are for, make issues appear early and fix them. I really would like to continune testing the N dev preview, but i need a app based on this lib urgently and it's very annoying to being able to use it. PLEASE start on fixing this problem as soon as possible.

[mikecousins]

Hopefully this gets prioritized soon. The ease for user's to install the latest dev preview has resulted in quite a few people not being able to run anything using your library.

[smarkovik]

Hello guys, any ideas when this will get in your pipeline? N preview has been out for a while and the apps which depend on sql cipher are essentially blocked.

[sjlombardo]

@smarkovik we just finished up a release of SQLCipher yesterday. Now that is out of the way, we'll be switching over to look at this issue as a priority.

[smarkovik]

Hello @sjlombardo, Not being pushy but just curious of your progress? any ETAs? could the community help ?

[developernotes]

Hi @smarkovik

We are actively working on this project and will make our changes available as soon as everything is working. Thanks!

[mvojjala]

Hi @sjlombardo

We are also in same situation. Not able use our app on Android N because of this issue. Just want to check with you if you have any update or tentative dates of a possible update.

Thanks

[sjlombardo]

@mvojjala we're making appreciable progress on this issue, but it does involve some fairly major changes. Keep an eye on this ticket, and/or at https://discuss.zetetic.net/c/sqlcipher for information about upcoming availability for testing.

[fuentespatrick]

I appreciate the effort and the update @sjlombardo. My team is similarly blocked, so the significant work is greatly appreciated!

[developernotes]

Hi folks,

We have just announced a beta of SQLCipher for Android with support for Android Developer N Preview here. Please take a look if you are interested in helping test out these changes. Thanks!

[smarkovik]

Awsome work guys !

[fuentespatrick]

Thanks, we'll give it a try!

[zokipirlo]

Tnx. I think it will work, but I have also a dependency on IOCipher. It's crashing while loading stlport_shared inside static block in VirtualFileSystem.java. I tried to remove it, but it's still loading it in iocipher. Is there any easy solution to make it work?

[developernotes]

Hi @zokipirlo

It sounds as if IOCipher may need to be rebuilt referencing the new version of SQLCipher for Android. IOCipher has publish build instructions on their README here.

[zokipirlo]

It works great. Everything looks fine. Thank you. I was having some problems with building IOCipher but that's another story :) Doesn't matter.

[developernotes]

Hello @zokipirlo

Thanks for the report, we are glad to hear it is working well for you!

[brodybits]

I am also really happy that you guys made the solution. I am especially happy that you guys removed the ICU and native setLocale parts, though it may be nice to have a version with ICU for those who may need it. I encounter 3 minor failures when I ran the SQLCipher test suite:

• Unicode (ICU) test
• Closed database test
• Invalid password test

The following change would the invalid password test:

--- a/src/main/java/net/zetetic/ZeteticApplication.java
+++ b/src/main/java/net/zetetic/ZeteticApplication.java
@@ -53,7 +53,10 @@ public class ZeteticApplication extends Application {
     public SQLiteDatabase createDatabase(File databaseFile){
         Log.i(TAG, "Entered ZeteticApplication::createDatabase");
         Log.i(TAG, "Before SQLiteDatabase.openOrCreateDatabase");
-        return SQLiteDatabase.openOrCreateDatabase(databaseFile, DATABASE_PASSWORD, null);
+        SQLiteDatabase database = SQLiteDatabase.openOrCreateDatabase(databaseFile, DATABASE_PASSWORD, null);
+        Log.i(TAG, "Populating the database");
+        database.rawExecSQL("CREATE TABLE IF NOT EXISTS MyTest (TestData)");
+        return database;
     }

     public void extractAssetToDatabaseDirectory(String fileName) throws IOException {

The closed database test fails because it tries the SQLiteDatabase.setLocale function which then throws an java.lang.UnsatisfiedLinkError with a "Native method not found" message. Considering the native_setLocale method is no longer implemented, I would favor fixing SQLiteDatabase.setLocale to just throw a runtime exception with a message that the method is no longer implemented.

[developernotes]

Hi @brodybits

Thank you for your feedback, it is much appreciated! Have you pulled down the latest changes to the SQLCipher for Android test suite? It should address those tests and also include a few new tests too.

[brodybits]

Have you pulled down the latest changes to the SQLCipher for Android test suite?

I missed it, I definitely like the changes to the test suite. I will give it a try sometime next week and let you know if I find anything else.

An extra point is that I wish you guys would consider supporting a version that something like BoringSSL which supports 64-bit builds out of the box.

[developernotes]

I missed it, I definitely like the changes to the test suite. I will give it a try sometime next week and let you know if I find anything else.

Sounds good.

An extra point is that I wish you guys would consider supporting a version that something like BoringSSL which supports 64-bit builds out of the box.

The 1.1.0 release of OpenSSL, currently scheduled for release on May 12th, 2016 will include support for x64.

[brodybits]

I just ran the test suite and it seems to run faster than before. Assuming that OpenSSL 1.1.0 is available next week will you release this change first or wait to include OpenSSL 1.1.0? (I would favor waiting for OpenSSL 1.1.0.)

[developernotes]

Hi @brodybits

Yes, we would like to include the OpenSSL 1.1.0 release in a future beta build. We would like to gather ample feedback from the community regarding these changes before we integrate them forward as a public release.

[zokipirlo]

Internal testing looks fine. I released an app update on Play Store with this version of Sqlcipher :) Will tell you soon if any crash reports will occur.

[brodybits]

I also find it pretty strange that I still cannot build this on Android API 23 and raised PR #230 to fix this.

[brodybits]

In the past I could do make init and it would automatically use the installed NDK build implementation. In the Android N preview I get a message that I have to define ANDROID_NDK_ROOT when building (I only have to define this for the make init step). I hope this will be documented.

Sorry to be a pain about this but I find it strange that I cannot build this with android-23 and raised #231 which I think fixes the build the right way. If not, I hope we can find another solution.

[developernotes]

In the past I could do make init and it would automatically use the installed NDK build implementation. In the Android N preview I get a message that I have to define ANDROID_NDK_ROOT when building (I only have to define this for the make init step).

The is due to building the OpenSSL libraries from the make init step. Previously we would build them locally and check them into source control. The aim with this branch was to streamline some of those efforts, so building the OpenSSL dependency once within make init was practical. The requirement to define ANDROID_NDK_ROOT was always present in the build-openssl-libraries.sh file; here is the relevant portion in the file found in the master branch.

[brodybits]

Ah now I understand. I hope this will be documented.

[developernotes]

Hi @brodybits

The documentation has been updated. Thanks

[mvojjala]

Hi @sjlombardo

Everything works great with the beta libraries in our app except for logs and I see that it already reported. Thank you for your effort. Any time line on when will be the final version released ?

Thanks

[sjlombardo]

@mvojjala thanks for letting us know things are working well for you. We haven't finalized a time frame yet, as we are still collecting feedback on the beta. We will likely release one or two more beta versions that will reduce the logging, and if the upcoming OpenSSL release works as planned incorporates a new version. Please keep an eye out, we'll be sure to announce upcoming releases on the site, and will make a note on this ticket as well.

[marcardar]

Is there an AAR package out there for those of us on slow connections...

[brodybits]

FYI make clean doesn't seem to work for me.

[brodybits]

While I am not in a position to participate in the official beta testing I have integrated a version with the android-n-preview changes in my Cordova plugin. I am using a custom version at https://github.com/litehelpers/android-database-sqlcipher-api-fix with the following additional fixes:

• Fixes to support API 23 as proposed in #231
• Working distclean target

[developernotes]

Hello @marcardar,

Is there an AAR package out there for those of us on slow connections...

No, if you prefer to not build from source you are welcome to request the beta builds through the support channel mentioned in this post, however they are distributed as a zip, not AAR. Keep in mind, cloning and initializing the android-n-preview branch is much smaller, and compiles much faster than the master branch.

[developernotes]

Hi @brodybits

FYI make clean doesn't seem to work for me.

What issue are you having with make clean?

[brodybits]

This is interesting: if I build the project successfully (or just finish make init successfully) then do make clean it seems to be OK. But if I stop make init in the middle or make init fails then do make clean I get the following (cut off after the make error):

cd /Users/brodybits/cordova-plugin/android-database-sqlcipher-api-fix/jni && \
    ndk-build clean
Android NDK: ERROR:/Users/brodybits/cordova-plugin/android-database-sqlcipher-api-fix/jni/Android.mk:static-libcrypto: LOCAL_SRC_FILES points to a missing file    
Android NDK: Check that /Users/brodybits/cordova-plugin/android-database-sqlcipher-api-fix/jni/../external/android-libs/armeabi/libcrypto.a exists  or that its path is correct   
/usr/local/Cellar/android-ndk/r11c/build/core/prebuilt-library.mk:45: *** Android NDK: Aborting    .  Stop.
make: [clean-ndk] Error 2 (ignored)

I would be happy if we just add a working distclean target that blows away all generated artifacts. For example: https://github.com/litehelpers/android-database-sqlcipher-api-fix/commit/b0fa9d3c8d7d0229fd8fc7424cdac7d8f9521e6e (please let me know if you want me to issue a PR)

P.S. I know most of my suggestions these days are "nits" as opposed to dealing with broken or missing functionality and understand you guys must be dealing with more important issues right now. I only raise these "nits" in hopes they will not be forgotten in the future.

[marcardar]

Thanks @developernotes . Unfortunately, I don't have an Android N device for testing so don't satisfy the requirements for obtaining the zip. I do have several pre-N devices though and satisfy all other items, so it's a shame you have that requirement.

Thanks very much for the suggestion to only clone that branch. I'm trying that now, so hopefully my connection can manage that!

[developernotes]

Hello @marcardar

Thank you for your interest, I've reached out to you directly.

[neerajgautam]

I am trying to launch my app on Android N Preview 3 , but getting following error on launching and app crashes. Can anyone please help!! I am stuck. Thanks in advance. APP_PACKAGE = application package name SO_FILE = Library file name

05-20 15:22:53.589: E/linker(8449): library "/system/lib/libjavacore.so" ("/system/lib/libjavacore.so") needed or dlopened by "/APP_PACKAGE/lib/arm/SO_FILE NAME.so" is not accessible for the namespace: [name="classloader-namespace", ld_library_paths="", default_library_paths="/APP_PACKAGE/lib/arm:/system/fake-libs:/APP_PACKAGE/base.apk!/lib/armeabi", permitted_paths="/data:/mnt/expand:/data/data/APP_PACKAGE"]

[powder366]

https://github.com/sqlcipher/sqlcipher/issues/158 Please fix.

[ghost]

The phone I am using is 6p, I got error as follows: java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.google.android.apps.talk-1/base.apk"],nativeLibraryDirectories=[/data/app/com.google.android.apps.talk-1/lib/arm64, /data/app/com.google.android.apps.talk-1/base.apk!/lib/arm64-v8a, /vendor/lib64, /system/lib64]]] couldn't find "libsqlcipher.so"

[brodybits]

The phone I am using is 6p, I got error as follows: java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/data/app/com.google.android.apps.talk-1/base.apk"],nativeLibraryDirectories=[/data/app/com.google.android.apps.talk-1/lib/arm64, /data/app/com.google.android.apps.talk-1/base.apk!/lib/arm64-v8a, /vendor/lib64, /system/lib64]]] couldn't find "libsqlcipher.so"

@lena8912 your app has one or more 64-bit libraries but this project does not include a 64-bit library which is why you get this error. Please be sure to remove all 64-bit libraries from your project's libs and from any dependencies that may be included.

[developernotes]

Hi @neerajgautam,

That error message doesn't appear to reference SQLCipher for Android. Did you have SQLCipher for Android previously integrated within your application?

[Pratyul]

Do we have an rough estimate on dates when the N support with be GA. It will help everyone in planning their releases.

[zokipirlo]

Just release it :) I did it almost a month ago and didn't receive a single crash. Even net.sqlcipher.database.SQLiteException: not an error: Could not register Android SQL functions. is gone from logs.