net.sqlcipher.database.SQLiteException ??? not error exception on SQLiteDatabase.dbOpen

kivsiak commented 9 years ago

I cannot found clear conditions to reproduce bug. But 2-5% my application starts are interrupted with this.

I have icudt46l.zip in assets. I've checked and found that /icu/icudt46l.dat is exists in my app data folder.

To be sure i run loadLibs() in application class. However sometimes(!) i got this.

net.sqlcipher.database.SQLiteException: not an error
   at net.sqlcipher.database.SQLiteDatabase.dbopen(SQLiteDatabase.java)
   at net.sqlcipher.database.SQLiteDatabase.<init>(SQLiteDatabase.java:1942)
   at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:875)
   at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:907)

Android 4.4.4 on nexus 7

developernotes commented 9 years ago

Hello @kivsiak

Do you only receive this on that specific device? You mention above that /icu/icudt46l.dat exists, do you actually mean /system/usr/icu/icudt46l.dat? Is loadLibs(…) running before you attempt to call SQLiteDatabase.openOrCreateDatabase?

kivsiak commented 9 years ago

Not a specific to device. Few devices with same problem.

/system/usr/icu/icudt46l.dat - No, mean /data/package.name/files/icu/icudt46l.dat

Yes i call loadLibs before any database operations. Worst of its not stable bug. According crashlitcs stats it happening in 2-5% of application runs. Not every one.

laminsk commented 9 years ago

I'm also getting the same error on nexus 5. Any help would be greatly appreciated!

developernotes commented 9 years ago

Hi @laminsk

Are you able to reliably reproduce the scenario on your Nexus 5. What version of Android in on your device?

laminsk commented 9 years ago

Thanks for the prompt reply. I have just managed to sort it out, basically removing and reinstalling the app fixed the problem. I'm running 4.4.2 and it was working alright until recently. I will keep testing and see how it goes. Many Thanks!

draak567 commented 9 years ago

We have a commercial app deployed to a few hundred thousand users, and every once in a while, an app install breaks with this problem. The only fix is to completely remove the APK from the device (clearing the data has no effect) and to reinstall it.

Interestingly, uninstalling the APK and reinstalling the same APK always fixes the issue.

Happens very rarely. I would say less than 0.1% of installs. We've seen it internally too.

jpuderer commented 9 years ago

I'm also seeing this issue, and I'm unable to reproduce it reliably.

marcardar commented 9 years ago

There is a problem with the SQLiteDatabase.loadICUData() method: https://github.com/sqlcipher/android-database-sqlcipher/blob/master/src/net/sqlcipher/database/SQLiteDatabase.java#L87

You should delete any incomplete icudt46l.dat file in the catch{} (and also throw an exception) and also include a finally{} block to close the streams.

Furthermore, you should synchronize{} (probably easiest to just synchronize the loadLibs() method) because otherwise if two threads call loadLibs() around the same time and the first one is in the middle of extracting the zip, then the second one will proceed with the partially extracted zip.

It could be that sometimes the user is left with a corrupt icudt46l.dat file which never gets corrected until uninstall-reinstall

This might also explain observation of @draak567 (that Clear Data has no effect) in the case of insufficient storage space or repeatedly extracting zip to same broken storage area.

developernotes commented 9 years ago

Hi @marcardar

Thanks for the suggestions, we will definitely look to adjust this accordingly.

brodybits commented 9 years ago

I also noticed the same issue in the following discussion on the new forum: https://discuss.zetetic.net/t/roots-of-the-problem-sqliteexception-not-an-error/586

For many error cases, jni/net_sqlcipher_database_SQLiteDatabase.cpp uses sqlite3_errmsg(handle) which tries to get the actual error message from the database handle. But there are certain cases where sqlite3_errmsg(handle) will return "not an error", as I found using https://www.google.com/#q=sqlite3+not+an+error and https://www.google.com/#q=sqlite3_errmsg+not+an+error:

Looking at these links, I cannot figure out how this could happen in the native dbopen function. But in https://github.com/android/platform_frameworks_base/blob/master/core/jni/android_database_SQLiteConnection.cpp I notice the following major differences in error reporting:

if sqlite3_open_v2() fails, they use the actual returned error code and an explicit message when raising an exception:

    int err = sqlite3_open_v2(path.string(), &db, sqliteFlags, NULL);
    if (err != SQLITE_OK) {
        throw_sqlite3_exception_errcode(env, err, "Could not open database");
        return 0;
    }

in the other cases, if they detect a failure they just add an explicit message to describe where it failed, for example:

    if ((sqliteFlags & SQLITE_OPEN_READWRITE) && sqlite3_db_readonly(db, NULL)) {
        throw_sqlite3_exception(env, db, "Could not open the database in read/write mode.");
        sqlite3_close(db);
        return 0;
    }

I recommend that we add better error reporting to this project to avoid this kind of confusion.

brodybits commented 9 years ago

Just issued PR #174 with fixes to show where SQLiteDatabase.dbopen is actually failing.

brodybits commented 9 years ago

Just reworked the changes in PR #175 to show where SQLiteDatabase.dbopen() is actually failing.

zokipirlo commented 8 years ago

I'm getting this exception on 3.3.0 build. No errors yet in 3.3.1 but it's only for a day in production, so it's too early to say it's fixed.

Caused by: net.sqlcipher.database.SQLiteException: not an error: Could not register Android SQL functions.
       at net.sqlcipher.database.SQLiteDatabase.dbopen(SQLiteDatabase.java)
       at net.sqlcipher.database.SQLiteDatabase.(SQLiteDatabase.java:2165)
       at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:1000)
       at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:1043)
       at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:132)

It happened on Galaxy Note4, Galaxy S4 Active, Sony Xperia M2 Aqua, Samsung S6 (SM-G920F), HTC One M8, Chromium ARC.

Did someone else received such an error? Is there any solution for that?

developernotes commented 8 years ago

Hi @zokipirlo

Are you using the community or commercial build of SQLCipher for Android? Are you able to reproduce this error, or are they intermittent on those devices?

zokipirlo commented 8 years ago

Hi, thanks for reply.

It's community build. No, I can't. I just see reports in crashlytics. And they happens almost every day. I will see what I could do to debug that problem.

Is there any debug version of sqlcipher with more logging?

developernotes commented 8 years ago

Hi @zokipirlo

Would you mind sharing the Android OS version from the respective devices you listed above that you have a crash report on? Further debugging would requiring native debugging of the source.

zokipirlo commented 8 years ago

No problem, what ever you need. Samsung S6 is 5.1.1. HTC is 5.0.1. No signs of root.

zokipirlo commented 8 years ago

There is one strange thing. I don't see any of this reports until recently, and it's in use more than a half year (in closed circle, it's still beta). Maybe it's not even a crash, could be just some error printing. Don't know.

developernotes commented 8 years ago

Hello @zokipirlo

Just to clarify, you haven't received any reports of crashes with SQLCipher for Android 3.3.1, and previous reports were all from 3.3.0?

zokipirlo commented 8 years ago

Yes. No crashes in 3.3.1 for now.

Ereza commented 8 years ago

Hi, I am using version 3.3.1 of the library and I always have the following crash on Nexus 5 (Android 5.1.1, build LMY48B) even when uninstalling and reinstalling the app. It's the first time I use the library so maybe it's a fault on my side.

09-21 10:03:06.336    3702-3702/********* E/AndroidRuntime﹕ FATAL EXCEPTION: main
    Process: *********, PID: 3702
    java.lang.RuntimeException: Unable to create application *********.MyApplication: net.sqlcipher.database.SQLiteException: not an error: Could not register Android SQL functions.
            at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4556)
            at android.app.ActivityThread.access$1500(ActivityThread.java:151)
            at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1364)
            at android.os.Handler.dispatchMessage(Handler.java:102)
            at android.os.Looper.loop(Looper.java:135)
            at android.app.ActivityThread.main(ActivityThread.java:5254)
            at java.lang.reflect.Method.invoke(Native Method)
            at java.lang.reflect.Method.invoke(Method.java:372)
            at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
            at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)
     Caused by: net.sqlcipher.database.SQLiteException: not an error: Could not register Android SQL functions.
            at net.sqlcipher.database.SQLiteDatabase.dbopen(Native Method)
            at net.sqlcipher.database.SQLiteDatabase.<init>(SQLiteDatabase.java:2165)
            at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:1000)
            at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:972)
            at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:1036)
            at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:1050)
            at *********.utils.MigrationUtils.importInitialDatabase(MigrationUtils.java:99)
            at *********.utils.MigrationUtils.migrateIfNeeded(MigrationUtils.java:31)
            at *********.MyApplication.onCreate(MyApplication.java:53)
            at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1012)
            at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4553)
            at android.app.ActivityThread.access$1500(ActivityThread.java:151)
            at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1364)
            at android.os.Handler.dispatchMessage(Handler.java:102)
            at android.os.Looper.loop(Looper.java:135)
            at android.app.ActivityThread.main(ActivityThread.java:5254)
            at java.lang.reflect.Method.invoke(Native Method)
            at java.lang.reflect.Method.invoke(Method.java:372)
            at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
            at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)

I can provide more info if needed.

Ereza commented 8 years ago

OK, it was indeed my fault. I had some code that was deleting all internal storage files after calling SQLiteOpenHelper.loadLibs(context);, which puts the ICU file on internal storage.

After changing that code to not remove ICU files, everything appears to work as expected.

developernotes commented 8 years ago

Hello @Ereza

We are glad to hear the issue is resolved on your end. Take care!

ilber commented 8 years ago

Hi @zokipirlo ,

I have your same problem on all the Android Devices prior Lollipop.

net.sqlcipher.database.SQLiteException: not an error: Could not register Android SQL functions. at net.sqlcipher.database.SQLiteDatabase.dbopen(Native Method) at net.sqlcipher.database.SQLiteDatabase.(SQLiteDatabase.java:2165) at net.sqlcipher.database.SQLiteDatabase.openDatabase(SQLiteDatabase.java:1000) at net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase(SQLiteDatabase.java:1043) at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:132) at net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase(SQLiteOpenHelper.java:99)

Do you solved your problems? I have the latest version of sqlcipher comunity edition. Any idea?

Thanks for the help/

Regards,

Ilber

ilber commented 8 years ago

No news here?? I have the community edition 3.3.1 and I have this problem on devices prior Android 5. I'm not deleting nothing like @Ereza but I still get this.

@developernotes do you have any idea?

ilber commented 8 years ago

After a lot of investigation I found the problem. It was with the ICU file. The file directory was not there when the app was loaded and the file directory was not created to when the loadLibs was called. Changing the directory to the app root it worked perfectly. Just if somebody has the same problem. call the loadLibs giving the root path of your app.

Regards,

Ilber

zokipirlo commented 8 years ago

I didn't see any problem since last update. But I did a lot of changes since than, so I don't know if it was fixed with library update or I had some other problem.

What I can tell you to check is:

if you use database as a singleton be sure to create it after you have a correct password (I had some problem that I try to use it with empty password; if you use it with cacheword check that with printing encryptionkey in logcat)
when destroying app be sure to clear singleton
if you have wipe data option be sure to clear singleton and delete database file
check if you have all necessary files in correct folders (icudt46l.zip must be in main projects assets folder)

developernotes commented 8 years ago

Hello @ilber

We are glad to hear you have resolved your issue. For clarification, what directory did not exist on your device before your call to loadLibs(…);?

ilber commented 8 years ago

the files dir of the app was not there and I don't know why the mkDir when saving the ICU file didn't create it. I pointed everything to the root path while loading the libs and everything started working on the old Android versions too.

developernotes commented 8 years ago

Hi @ilber

Interesting, it should create it if the folder does not exist, did you see a log message from an exception being thrown? Can you run the SQLCipher for Android test suite successfully?

ilber commented 8 years ago

Yes the test suite is running without problems. I saw the code and it should create it but was not created for some reasons. But changing to another root, everything works fine. Maybe it was something else. I will investigate more on that and check whats going on. Thank you for your amazing job.

zokipirlo commented 8 years ago

I found one issue today on Genymotion emulator. I was getting crashes again Caused by: net.sqlcipher.database.SQLiteException: not an error: Could not register Android SQL functions..

Problem was that /system/usr/icu/icudt46l.dat doesn't exists and copy of file icudt46l.dat wasn't successful. So icudt46l.dat file was created in internal storage but size was 0. I suggest that in loadICUData method there should be check if icuDataFile has correct length or at least greater than zero.

Maybe it was just some crazy edge case and buggy emulator but I think it wouldn't do any harm if that check is included.

developernotes commented 8 years ago

Hi @zokipirlo

Is that a behavior you can consistently reproduce with the Genymotion emulator? What emulator were you using?

zokipirlo commented 8 years ago

No, I can't. Moto X 4.4.4. I tried a few times now but it didn't repeat. It must be some strange coincidence.

McPo commented 7 years ago

What was the reasoning for this issue being closed.

Still experiencing issues with 3.4.0 in the field and am unable to reproduce.

developernotes commented 7 years ago

Hi @McPo

There were various changes merged in to help resolve issues around loading of the ICU dat file, both @zokipirlo and @Ereza reported they are no longer seeing the issue following those changes. Are you able to reproduce the error within the SQLCipher for Android test suite? Also, the android-n-preview branch which we plan to release in the near term removes ICU entirely from the library so it will not likely be a relevant issue.

McPo commented 7 years ago

We have only seen it once with a device in our possession, and havn't been able to replicate it since. Had to clear the data to resolve the issue. As such I unfortunately have no steps to reproduce it in the test suite. I'll keep an eye out and update this issue if I come across any new information. It may simply be a case of waiting until the android-n-preview is released. Thanks.

vincentskooi commented 7 years ago

Hi, I am using "net.zetetic:android-database-sqlcipher:3.4.0", I see this issue from my end and it seem to fail consistently but i found out that it happened during a fresh app installation where both my UnitTest and Application Class are trying to access the singleton DBHandler for the first time (to create DB). I added synchronization on my Singleton DBHandler and error goes away..

developernotes commented 7 years ago

Hello @McPo, @vincentskooi, @kivsiak,

Please try the latest release of SQLCipher for Android, 3.5.1 and let us know if you are still seeing this issue.

vincentskooi commented 7 years ago

Hi, I have upgraded to 3.5.1. I don't get this error but I get a different exception: E/GreenDaoDBHandler.GreenDaoDBHandler(): Database Exception: file is encrypted or is not a database: BEGIN EXCLUSIVE; | stack:net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: BEGIN EXCLUSIVE;

The issue I have is most likely synchronization issue, I put back the syncrhonized on my Singleton DBHandler and it is working fine now.

developernotes commented 6 years ago

Hi @3arl0ck

along side this, app also crashes with java.lang.UnsatisfiedLinkError:

That would suggest your application is not properly including the native libraries, or you are stripping them from the final APK via ProGuard.

iamriajul commented 5 years ago

I'm also getting the same error on Nexus 5X. Any help would be greatly appreciated!

developernotes commented 5 years ago

Hi @iamriajul

A SQLiteException is a generic error, we will need you to provide more information in order to assist in debugging the issue. The issue above has been closed for almost 4 years. I would recommend creating a new GitHub Issue where you provide additional information regarding the source code that is executed that causes an error, any additional reproduction steps and the Android OS version. Without additional information, we won't be able to assist you.

sqlcipher / android-database-sqlcipher

net.sqlcipher.database.SQLiteException ??? not error exception on SQLiteDatabase.dbOpen #139