Unicode data not being inserted

[iambibhas]

I'm trying to insert some unicode strings to the db like this -

questionId = questionObject.getInt("id");
                text = questionObject.getString("text");
                text_kn = questionObject.getString("text_kn");
                Log.d(this.toString(), text_kn);

                Question question = new Question()
                        .setId(questionId)
                        .setText(text)
                        .setTextKn(text_kn);

                db.insertWithId(question);
                Log.d(this.toString(), question.getTextKn());

insertWithId() simply uses REPLACE as conflict algorithm. It works for everything else.

In the above code, the first log shows the strings, the second log also shows the string. But when I inspect the db or print the record in another activity, that text_kn column is empty. Not sure what's happening.

I do get some warnings that my db connection is leaking, which I'm fixing right now using singleton pattern. Not sure if that's the reason.

Any idea?

Update: I fixed the leaking connection issue using singleton, but the strings are still not being inserted.

[sbosley]

Not sure what could be going wrong here. We do explicitly test for unicode strings in our test suite as seen here: testReadUnicodeStrings(), so it should be working. If you can send us a more complete test case that reproduces the problem you're seeing we can take a look.

[jdkoren]

@iambibhas out of curiosity, can you post one of the strings that appears not to have been stored?

[iambibhas]

@jdkoren this is one -

ಕಳೆದ ತಿಂಗಳಲ್ಲಿ ನಿಮ್ಮ ಮಗುವಿನ ಶಾಲೆಗೆ ಭೇಟಿ ನೀಡಿದ್ದೀರಾ?

Or

\u0cb6\u0cbe\u0cb2\u0cc6\u0caf\u0cb2\u0ccd\u0cb2\u0cbf \u0cac\u0cb3\u0cb8\u0cb2\u0cc1 \u0caf\u0ccb\u0c97\u0ccd\u0caf \u0caa\u0ccd\u0cb0\u0ca4\u0ccd\u0caf\u0cc7\u0c95 \u0cb9\u0cc1\u0ca1\u0cc1\u0c97\u0cbf\u0caf\u0cb0 \u0cb6\u0ccc\u0c9a\u0cbe\u0cb2\u0caf\u0cb5\u0cbf\u0ca6\u0cc6\u0caf\u0cc7?

[iambibhas]

It's definitely not being inserted in db, I fetched the record right after it was inserted and that property was empty -

db.insertWithId(question);
Log.d(this.toString(), question.getTextKn());
Question q2 = db.fetch(Question.class, questionId);
Log.d(this.toString(), q2.getTextKn());

Second log throws error as getTextKn() returns null.

[iambibhas]

Ok. I think I figured it out. There was a duplicate record in the db with the same _id. I think sqlite didn't know which one to replace. This looks like a caveat of playing around with _id. Not sure though how the duplicate entry got inserted there. I'll keep an eye open for that and update it here if I find out. Thanks for the help! You guys built something really good. :) Hoping to get enough used to this library so that I can also contribute someday. Cheers.

[sbosley]

Ok cool, glad it seems to be (mostly) resolved! I am a bit surprised that REPLACE didn't work as expected with _id -- if you want to post your insertWithId() code I can take a look to see if there's anything odd going on there, and do some additional experiments to see if there are any other unexpected behaviors of using the row id like that we need to be aware of. We're in the process of reworking some of our primary key support, so in case there's something odd going on there we're not aware of it might be helpful to learn it now rather than later.

[sbosley]

Hey @iambibhas, quick follow up. I wrote the following test case just to make sure I wasn't missing anything:

public void testInsertWithIdUnicode() {
    String unicode = "\u0cb6\u0cbe\u0cb2\u0cc6\u0caf\u0cb2\u0ccd\u0cb2\u0cbf \u0cac\u0cb3\u0cb8\u0cb2\u0cc1 "
            + "\u0caf\u0ccb\u0c97\u0ccd\u0caf \u0caa\u0ccd\u0cb0\u0ca4\u0ccd\u0caf\u0cc7\u0c95 "
            + "\u0cb9\u0cc1\u0ca1\u0cc1\u0c97\u0cbf\u0caf\u0cb0 "
            + "\u0cb6\u0ccc\u0c9a\u0cbe\u0cb2\u0caf\u0cb5\u0cbf\u0ca6\u0cc6\u0caf\u0cc7?";

    // Insert a model with no unicode string
    Thing test = new Thing().setId(1).setFoo("Test1");
    assertTrue(database.insertWithId(test));
    assertEquals(1, test.getId());
    assertEquals(1, database.countAll(Thing.class));

    // Check that this initial row with _id = 1 was inserted correctly
    Thing fetched1 = database.fetch(Thing.class, test.getId());
    assertEquals(1, fetched1.getId());
    assertEquals("Test1", fetched1.getFoo());

    // Use insertWithId where _id = 1 to replace the existing model with one containing the unicode string
    Thing test2 = new Thing().setId(1).setFoo(unicode);
    assertTrue(database.insertWithId(test2));
    assertEquals(1, test2.getId());
    assertEquals(1, database.countAll(Thing.class));

    // Fetch the row again from the database and check if the unicode was read correctly
    Thing fetched2 = database.fetch(Thing.class, test2.getId());
    assertEquals(1, fetched2.getId());
    assertEquals(unicode, fetched2.getFoo());
}

insertWithId() was a simple wrapper for insertRow(item, ConflictAlgorithm.REPLACE).

When I ran this unit test it did pass as expected, indicating that REPLACE was working correctly even when using _id and insertWithId. I'm guessing that there may be something else going on with your setup that is causing the odd behavior you're seeing. If you want to post more of your code we might be able to help you figure out why things weren't behaving as expected for you.

[iambibhas]

@sbosley pretty sure something was wrong with my data updating code. I haven't found out yet. But it was not because of volley I think. Thanks for adding the test case just in case. You can close this now as it works as expected..

[sbosley]

Ok cool, let us know if you run into other issues!