Exception on API level 28

[mehdi-salehi]

Expected Behavior

Working on all API levels.

Actual Behavior

We have many Exceptions on API level 28 but other versions are working properly.

Steps to Reproduce

We can not produce this exception in debug Env. We are migrating to encrypted DB and Crash reports are too high on Android 9 devices(like HUAWEI Y9 2019);

first bug: java.lang.NullPointerException: Attempt to invoke virtual method 'int net.sqlcipher.database.SQLiteDatabase.getVersion()' on a null object reference

com.SQLiteAssetHelper.getWritableDatabase SQLiteAssetHelper.java:141 com.DatabaseHelper. DatabaseHelper.java:70 com.DatabaseHelper.getInstance DatabaseHelper.java:80 com.co.tools.sqlite.DB_Configuration.set DB_Configuration.java com.co.mainActivity.onCreate m.java:99 android.app.Activity.performCreate Activity.java:7458 android.app.Activity.performCreate Activity.java:7448 android.app.Instrumentation.callActivityOnCreate Instrumentation.java:1286 android.app.ActivityThread.performLaunchActivity ActivityThread.java:3409 android.app.ActivityThread.handleLaunchActivity ActivityThread.java:3614 android.app.servertransaction.LaunchActivityItem.execute LaunchActivityItem.java:86 android.app.servertransaction.TransactionExecutor.executeCallbacks TransactionExecutor.java:108 android.app.servertransaction.TransactionExecutor.execute TransactionExecutor.java:68 android.app.ActivityThread$H.handleMessage ActivityThread.java:2199 android.os.Handler.dispatchMessage Handler.java:112 android.os.Looper.loop Looper.java:216 android.app.ActivityThread.main ActivityThread.java:7625 java.lang.reflect.Method.invoke Method.java com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run RuntimeInit.java:524 com.android.internal.os.ZygoteInit.main ZygoteInit.java:987

second bug: net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory SQLiteDatabase.java, line 2004 java.lang.IllegalStateException: database not open

Third bug: net.sqlcipher.database.SQLiteDatabase.execSQL SQLiteDatabase.java, line 2439 net.sqlcipher.database.SQLiteDatabaseCorruptException: database disk image is malformed: insert into help ...

net.sqlcipher.database.SQLiteDatabase.native_execSQL SQLiteDatabase.java:-2 net.sqlcipher.database.SQLiteDatabase.execSQL SQLiteDatabase.java:2439 com.DatabaseHelper.insertHelp DatabaseHelper.java:3348

SQLCipher version (can be identified by executing PRAGMA cipher_version;): 4.4.3 community

SQLCipher for Android version: implementation "net.zetetic:android-database-sqlcipher:4.5.2" implementation "androidx.sqlite:sqlite:2.2.0"

net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory SQLiteDatabase.java:2004 net.sqlcipher.database.SQLiteDatabase.rawQuery SQLiteDatabase.java:1902 com.DatabaseHelper.getConfiguration DatabaseHelper.java:104

Are you able to reproduce this issue within the SQLCipher for Android test suite? NO

Note: If you are not posting a specific issue for the SQLCipher library, please post your question to the SQLCipher discuss site. Thanks!

[developernotes]

Hello @mehdi-salehi,

Thanks for reaching out. In the future, please consider posting these as separate bug reports, it will simplify following any discourse regarding a specific issue.

We have many Exceptions on API level 9 but other versions are working properly.

SQLCipher for Android is supported on API 21 and up.

first bug: java.lang.NullPointerException: Attempt to invoke virtual method 'int net.sqlcipher.database.SQLiteDatabase.getVersion()' on a null object reference

com.SQLiteAssetHelper.getWritableDatabase SQLiteAssetHelper.java:141 com.DatabaseHelper. DatabaseHelper.java:70 com.DatabaseHelper.getInstance DatabaseHelper.java:80 com.co.tools.sqlite.DB_Configuration.set DB_Configuration.java com.co.mainActivity.onCreate m.java:99

This is a null reference exception. It would be best to investigated the code within the SQLiteAssetHelper class to see what invariants are not holding true. Without seeing the code it is difficult to say what the cause may be.

second bug: net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory SQLiteDatabase.java, line 2004 java.lang.IllegalStateException: database not open

What code within the application is being invoked that causes this behavior?

Third bug: net.sqlcipher.database.SQLiteDatabase.execSQL SQLiteDatabase.java, line 2439 net.sqlcipher.database.SQLiteDatabaseCorruptException: database disk image is malformed: insert into help ...

net.sqlcipher.database.SQLiteDatabase.native_execSQL SQLiteDatabase.java:-2 net.sqlcipher.database.SQLiteDatabase.execSQL SQLiteDatabase.java:2439 com.DatabaseHelper.insertHelp DatabaseHelper.java:3348

Are you able to extract the database off the device to see if you are able to inspect the file further? Please try running the following commands against the database file when this situation occurs:

• PRAGMA cipher_integrity_check
• PRAGMA integrity_check

SQLCipher version (can be identified by executing PRAGMA cipher_version;): 4.4.3 community

SQLCipher for Android version: implementation "net.zetetic:android-database-sqlcipher:4.5.2" implementation "androidx.sqlite:sqlite:2.2.0"

You are reporting two different versions of the library. Are you experiencing these issues on different applications?

Unfortunately, without providing additional details regarding what the applications are doing with the SQLCipher API it is difficult to speculate what may be causing some of these issues. If you are able to provide further details, please consider:

• Application interaction/code with SQLCipher API
• Whether the database file(s) exists or not
• Whether the database file is encrypted or plaintext
• What error checks have been put in place
• Other third-party API's interfacing with SQLCipher
• Are different versions of SQLCipher in use when clients receive an error

[mehdi-salehi]

Android 9 Api 28

Migrating from plaintext to encrypted

implementation "net.zetetic:android-database-sqlcipher:4.5.2" implementation "androidx.sqlite:sqlite:2.2.0"

local device: PRAGMA cipher_version = 4.5.2 community PRAGMA integrity_check = ok PRAGMA cipher_integrity_check = 0 row

We can not access DB file on crashed Devices; Things are OK on other API levels;

public class SQLiteAssetHelper extends SQLiteOpenHelper {

    private static final String TAG = SQLiteAssetHelper.class.getSimpleName();
    private static final String ASSET_DB_PATH = "databases";

    private final Context mContext;
    private final String mName;
    private final CursorFactory mFactory;
    private final int mNewVersion;

    private SQLiteDatabase mDatabase = null;
    private boolean mIsInitializing = false;

    private String mDatabasePath;

    private String mAssetPath;

    private String mUpgradePathFormat;

    private int mForcedUpgradeVersion = 0;

    public SQLiteAssetHelper(Context context, String name, String storageDirectory, CursorFactory factory, int version) {
        super(context, name, factory, version);

        if (version < 1) throw new IllegalArgumentException("Version must be >= 1, was " + version);
        if (name == null) throw new IllegalArgumentException("Database name cannot be null");

        mContext = context;
        mName = name;
        mFactory = factory;
        mNewVersion = version;

        mAssetPath = ASSET_DB_PATH + "/" + name;
        if (storageDirectory != null) {
            mDatabasePath = storageDirectory;
        } else {
            mDatabasePath = context.getApplicationInfo().dataDir + "/databases";
        }
        mUpgradePathFormat = ASSET_DB_PATH + "/" + name + "_upgrade_%s-%s.sql";
    }

    public SQLiteAssetHelper(Context context, String name, CursorFactory factory, int version) {
        this(context, name, null, factory, version);
    }

    @Override
    public synchronized SQLiteDatabase getWritableDatabase(String pass) {
        if (mDatabase != null && mDatabase.isOpen() && !mDatabase.isReadOnly()) {
            return mDatabase;  // The database is already open for business
        }

        if (mIsInitializing) {
            throw new IllegalStateException("getWritableDatabase called recursively");
        }

        // If we have a read-only database open, someone could be using it
        // (though they shouldn't), which would cause a lock to be held on
        // the file, and our attempts to open the database read-write would
        // fail waiting for the file lock.  To prevent that, we acquire the
        // lock on the read-only database, which shuts out other users.

        boolean success = false;
        SQLiteDatabase db = null;
        //if (mDatabase != null) mDatabase.lock();
        try {
            mIsInitializing = true;
            //if (mName == null) {
            //    db = SQLiteDatabase.create(null);
            //} else {
            //    db = mContext.openOrCreateDatabase(mName, 0, mFactory);
            //}
            db = createOrOpenDatabase(false, pass);

            int version = db.getVersion();

            // do force upgrade
            if (version != 0 && version < mForcedUpgradeVersion) {
                db = createOrOpenDatabase(true, pass);
                db.setVersion(mNewVersion);
                version = db.getVersion();
            }

            if (version != mNewVersion) {
                db.beginTransaction();
                try {
                    if (version == 0) {
                        onCreate(db);
                    } else {
                        if (version > mNewVersion) {
                            Log.w(TAG, "Can't downgrade read-only database from version " +
                                    version + " to " + mNewVersion + ": " + db.getPath());
                        }
                        onUpgrade(db, version, mNewVersion);
                    }
                    db.setVersion(mNewVersion);
                    db.setTransactionSuccessful();
                } finally {
                    db.endTransaction();
                }
            }

            onOpen(db);
            success = true;
            return db;
        } finally {
            mIsInitializing = false;
            if (success) {
                if (mDatabase != null) {
                    try {
                        mDatabase.close();
                    } catch (Exception e) {
                    }
                    //mDatabase.unlock();
                }
                mDatabase = db;
            } else {
                //if (mDatabase != null) mDatabase.unlock();
                if (db != null) db.close();
            }
        }

    }

    @Override
    public synchronized SQLiteDatabase getReadableDatabase(String pass) {
        if (mDatabase != null && mDatabase.isOpen()) {
            return mDatabase;  // The database is already open for business
        }

        if (mIsInitializing) {
            throw new IllegalStateException("getReadableDatabase called recursively");
        }

        try {
            return getWritableDatabase(pass);
        } catch (SQLiteException e) {
            if (mName == null) throw e;  // Can't open a temp database read-only!
            Log.e(TAG, "Couldn't open " + mName + " for writing (will try read-only):", e);
        }

        SQLiteDatabase db = null;
        try {
            mIsInitializing = true;
            String path = mContext.getDatabasePath(mName).getPath();
            db = SQLiteDatabase.openDatabase(path, pass, mFactory, SQLiteDatabase.OPEN_READONLY);
            if (db.getVersion() != mNewVersion) {
                throw new SQLiteException("Can't upgrade read-only database from version " +
                        db.getVersion() + " to " + mNewVersion + ": " + path);
            }

            onOpen(db);
            Log.w(TAG, "Opened " + mName + " in read-only mode");
            mDatabase = db;
            return mDatabase;
        } finally {
            mIsInitializing = false;
            if (db != null && db != mDatabase) db.close();
        }
    }

    @Override
    public synchronized void close() {
        if (mIsInitializing) throw new IllegalStateException("Closed during initialization");

        if (mDatabase != null && mDatabase.isOpen()) {
            mDatabase.close();
            mDatabase = null;
        }
    }

    @Override
    public final void onConfigure(SQLiteDatabase db) {
        // not supported!
    }

    @Override
    public final void onCreate(SQLiteDatabase db) {
        // do nothing - createOrOpenDatabase() is called in
        // getWritableDatabase() to handle database creation.
    }

    @Override
    public void onUpgrade(SQLiteDatabase db, int oldVersion, int newVersion) {

        Log.w(TAG, "Upgrading database " + mName + " from version " + oldVersion + " to " + newVersion + "...");

        ArrayList<String> paths = new ArrayList<String>();
        getUpgradeFilePaths(oldVersion, newVersion - 1, newVersion, paths);

        if (paths.isEmpty()) {
            Log.e(TAG, "no upgrade script path from " + oldVersion + " to " + newVersion);
            throw new SQLiteAssetException("no upgrade script path from " + oldVersion + " to " + newVersion);
        }

        Collections.sort(paths, new VersionComparator());
        for (String path : paths) {
            try {
                Log.w(TAG, "processing upgrade: " + path);
                InputStream is = mContext.getAssets().open(path);
                String sql = Utils.convertStreamToString(is);
                if (sql != null) {
                    List<String> cmds = Utils.splitSqlScript(sql, ';');
                    for (String cmd : cmds) {
                        //Log.d(TAG, "cmd=" + cmd);
                        if (cmd.trim().length() > 0) {
                            db.execSQL(cmd);
                        }
                    }
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

        Log.w(TAG, "Successfully upgraded database " + mName + " from version " + oldVersion + " to " + newVersion);

    }

    @Override
    public final void onDowngrade(SQLiteDatabase db, int oldVersion, int newVersion) {
        // not supported!
    }

    @Deprecated
    public void setForcedUpgradeVersion(int version) {
        setForcedUpgrade(version);
    }

    public void setForcedUpgrade(int version) {
        mForcedUpgradeVersion = version;
    }

    public void setForcedUpgrade() {
        setForcedUpgrade(mNewVersion);
    }

    private SQLiteDatabase createOrOpenDatabase(boolean force, String pass) throws SQLiteAssetException {
        // test for the existence of the db file first and don't attempt open
        // to prevent the error trace in log on API 14+
        SQLiteDatabase db = null;
        File file = new File(mDatabasePath + "/" + mName);
        if (file.exists()) {
            db = returnDatabase(pass);
        }
        //SQLiteDatabase db = returnDatabase();

        if (db != null) {
            // database already exists
            if (force) {
                Log.w(TAG, "forcing database upgrade!");
                copyDatabaseFromAssets();
                db = returnDatabase(pass);
            }
            return db;
        } else {
            // database does not exist, copy it from assets and return it
            copyDatabaseFromAssets();
            db = returnDatabase(pass);
            return db;
        }
    }

    private SQLiteDatabase returnDatabase(String pass) {
        try {
            SQLiteDatabase db = SQLiteDatabase.openDatabase(mDatabasePath + "/" + mName, pass, mFactory, SQLiteDatabase.OPEN_READWRITE);
            Log.i(TAG, "successfully opened database " + mName);
            return db;
        } catch (SQLiteException e) {
            Log.w(TAG, "could not open database " + mName + " - " + e.getMessage());
            return null;
        }
    }

    private void copyDatabaseFromAssets() throws SQLiteAssetException {
        Log.w(TAG, "copying database from assets...");

        String path = mAssetPath;
        String dest = mDatabasePath + "/" + mName;
        InputStream is;
        boolean isZip = false;

        try {
            // try uncompressed
            is = mContext.getAssets().open(path);
        } catch (IOException e) {
            // try zip
            try {
                is = mContext.getAssets().open(path + ".zip");
                isZip = true;
            } catch (IOException e2) {
                // try gzip
                try {
                    is = mContext.getAssets().open(path + ".gz");
                } catch (IOException e3) {
                    SQLiteAssetException se = new SQLiteAssetException("Missing " + mAssetPath + " file (or .zip, .gz archive) in assets, or target folder not writable");
                    se.setStackTrace(e3.getStackTrace());
                    throw se;
                }
            }
        }

        try {
            File f = new File(mDatabasePath + "/");
            if (!f.exists()) {
                f.mkdir();
            }
            if (isZip) {
                ZipInputStream zis = Utils.getFileFromZip(is);
                if (zis == null) {
                    throw new SQLiteAssetException("Archive is missing a SQLite database file");
                }
                Utils.writeExtractedFileToDisk(zis, new FileOutputStream(dest));
            } else {
                Utils.writeExtractedFileToDisk(is, new FileOutputStream(dest));
            }

            Log.w(TAG, "database copy complete");

        } catch (IOException e) {
            SQLiteAssetException se = new SQLiteAssetException("Unable to write " + dest + " to data directory");
            se.setStackTrace(e.getStackTrace());
            throw se;
        }
    }

    private InputStream getUpgradeSQLStream(int oldVersion, int newVersion) {
        String path = String.format(mUpgradePathFormat, oldVersion, newVersion);
        try {
            return mContext.getAssets().open(path);
        } catch (IOException e) {
            Log.w(TAG, "missing database upgrade script: " + path);
            return null;
        }
    }

    private void getUpgradeFilePaths(int baseVersion, int start, int end, ArrayList<String> paths) {

        int a;
        int b;

        InputStream is = getUpgradeSQLStream(start, end);
        if (is != null) {
            String path = String.format(mUpgradePathFormat, start, end);
            paths.add(path);
            //Log.d(TAG, "found script: " + path);
            a = start - 1;
            b = start;
            is = null;
        } else {
            a = start - 1;
            b = end;
        }

        if (a < baseVersion) {
            return;
        } else {
            getUpgradeFilePaths(baseVersion, a, b, paths); // recursive call
        }

    }

    @SuppressWarnings("serial")
    public static class SQLiteAssetException extends SQLiteException {

        public SQLiteAssetException() {
        }

        public SQLiteAssetException(String error) {
            super(error);
        }
    }

}

[developernotes]

Hi @mehdi-salehi,

It is difficult for us to speculate as to what the application is doing given the information that has been provided. The stack trace line below:

com.SQLiteAssetHelper.getWritableDatabase SQLiteAssetHelper.java:141

lines up with a log statement in the code sample you provided, this is likely due to the import statements being omitted. It appears that your application may attempt to extract a database file from a zip included as an asset within the application. Is that code being invoked prior to your crash scenario? Additionally, there are places where your application code does not check for a null database connection instance before invoking the call to getVersion(). Our recommendation would be to review the application code further in order to understand the scenario(s) in which an error is occurring. Additionally, you should consider adding additional guards to the application code to handle potential error states. If you are able to identify a reproducible error in the SQLCipher for Android library, we would love to look into the details further.

[mehdi-salehi]

Thanks for your comments.