m_iter Null Pointer Crash in MMICUTokenizer::nextToken when Inserting empty-string column value into FTS Virtual Table

[sooglejay]

The language of WCDB

Java

The version of WCDB

2.1.8 (master branch)

The platform of WCDB

Android

The installation of WCDB

Git clone

What's the issue?

I’m encountering a consistent crash in the MMICUTokenizer::nextToken function due to a null m_iter pointer when using WCDB. This occurs every time I run the following code:

database?.addTokenizer(BuiltinTokenizer.MMICU)
database?.runTransaction(Transaction {
    // Create the virtual table
    database?.execute("CREATE VIRTUAL TABLE IF NOT EXISTS jwFTS_12_ha USING fts4(tokenize=mmicu, fts_remark_name, fts_nick_name, fts_user_id);")

    // insert data
    val cv = ContentValues()
    cv.put("rowid", "94444448")
    cv.put("fts_remark_name", "")
    cv.put("fts_nick_name", "测试 数学 化学 ")
    cv.put("fts_user_id", "52222221 42222221")

    insertWithOnConflict("jwFTS_12_ha", null, cv, SQLiteDatabase.CONFLICT_REPLACE)
    true
})

// insertWithOnConflict is the function of com.tencent.wcdb.compat.SQLiteDatabase, converted into kotlin like below:
fun insertWithOnConflict(
    table: String?, nullColumnHack: String?,
    initialValues: ContentValues?, conflictAlgorithm: Int
): Long {
    val columns: Array<String?>
    val bindArgs: Array<Any?>
    val size =
        if (initialValues != null && initialValues.size() > 0) initialValues.size() else 0
    if (size > 0) {
        bindArgs = arrayOfNulls(size)
        columns = arrayOfNulls(size)
        var i = 0
        for (colName in initialValues!!.keySet()) {
            columns[i] = colName
            bindArgs[i++] = initialValues[colName]
        }
    } else {
        columns = arrayOf(nullColumnHack)
        bindArgs = arrayOf(null)
    }
    val insert = StatementInsert().insertInto(table!!)
    when (conflictAlgorithm) {
        SQLiteDatabase.CONFLICT_NONE -> {}
        SQLiteDatabase.CONFLICT_ROLLBACK -> insert.orRollback()
        SQLiteDatabase.CONFLICT_ABORT -> insert.orAbort()
        SQLiteDatabase.CONFLICT_FAIL -> insert.orFail()
        SQLiteDatabase.CONFLICT_IGNORE -> insert.orIgnore()
        SQLiteDatabase.CONFLICT_REPLACE -> insert.orReplace()
    }
    insert.columns(*columns).valuesWithBindParameters(bindArgs.size)
    database!!.getHandle(true).use { handle ->
        val stmt: PreparedStatement = handle.preparedWithMainStatement(insert)
        for (i in bindArgs.indices) {
            stmt.bindValue(Value(bindArgs[i]), i + 1)
        }
        do {
            stmt.step()
        } while (!stmt.isDone)
        stmt.finalizeStatement()
        return handle.getLastInsertedRowId()
    }
}

[sooglejay]

I have a new discovery regarding the crash issue. It appears that the problem arises specifically when the field is set to an empty string. When I replace cv.put("fts_remark_name", "") with a non-empty value like cv.put("fts_remark_name", "wewe"), the insertion works without any issues. but why and how to fix this crash?

[Qiuwen-chen]

Fixed in the master branch