DATABASE: Assertion failed: docID && revID && sequence > _lastSequence

[ddelapaz]

08-26 09:24:38.192 15426-19350/cb.cheneytracker E/CouchbaseLite/DATABASE: Assertion failed: docID && revID && sequence > _lastSequence (/home/couchbase/jenkins/workspace/couchbase-lite-android-edition-build/couchbase-lite-android-ee/libs/couchbase-lite-android/libs/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:113, in documentChanged)/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so litecore::SequenceTracker::documentChanged(fleece::alloc_slice const&, fleece::alloc_slice const&, unsigned long, unsigned long)/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so c4Internal::Database::documentSaved(c4Internal::Document*)/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so c4Internal::TreeDocument::save(unsigned int)/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so c4Internal::TreeDocument::saveNewRev(C4DocPutRequest const&, litecore::Rev const*, bool)/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so c4Internal::TreeDocument::putNewRevision(C4DocPutRequest const&)/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so c4doc_put/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so c4doc_update/data/app/cb.cheneytracker-2/lib/arm64/libLiteCoreJNI.so Java_com_couchbase_litecore_C4Document_update2/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/app/cb.cheneytracker-2/oat/arm64/base.odex oatexec/data/dalvik-cache/arm64/system@framework@com.symbol.emdk.jar@classes.dex oatexec

08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out: com.couchbase.lite.CouchbaseLiteException: assertion failed
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.CBLStatus.convertException(CBLStatus.java:51) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.CBLStatus.convertException(CBLStatus.java:55) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.AbstractDatabase.save(AbstractDatabase.java:1239) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.AbstractDatabase.save(AbstractDatabase.java:295) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.Database.save(Database.java:27) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.AbstractDatabase.save(AbstractDatabase.java:274) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.Database.save(Database.java:27) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at cb.cheneytracker.util.DBHelper.PopulateDocuments(DBHelper.java:159) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at cb.cheneytracker.Core.SaveCadecGlobalObjectToDevice(Core.java:1401) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at cb.cheneytracker.DeliveryScreenActivity.IncreaseOrDecreaseScanProcess(DeliveryScreenActivity.java:531) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at cb.cheneytracker.DeliveryScreenActivity.ProcessReceivedData(DeliveryScreenActivity.java:624) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at cb.cheneytracker.DeliveryScreenActivity.onData(DeliveryScreenActivity.java:693) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.symbol.emdk.barcode.DataCallbackInternal$DataCallbackThread.run(DataCallbackInternal.java:80) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out: Caused by: com.couchbase.litecore.LiteCoreException: assertion failed
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.litecore.C4Document.update2(Native Method) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.litecore.C4Document.update(C4Document.java:160) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.AbstractDatabase.save(AbstractDatabase.java:1283) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   at com.couchbase.lite.AbstractDatabase.save(AbstractDatabase.java:1237) ~[na:0.0]
08-26 09:24:38.206 15426-19350/cb.cheneytracker I/System.out:   ... 10 common frames omitted

I'm using com.couchbase.lite:couchbase-lite-android:2.5.3 on Android OS v 6.0.1

    compileSdkVersion 29
    buildToolsVersion '28.0.3'
    configurations { all*.exclude group: 'com.android.support', module: 'support-v7' }
    defaultConfig {
        applicationId "cb.cheneytracker"
        minSdkVersion 23
        targetSdkVersion 29
        multiDexEnabled true
    }

My current implementation is a bit different because I have so many changes I would need to make in order to utilize the query system in Couchbase Lite, so for now I'm simply serializing and deserializing data and saving to database.

The exception appears to take place while saving constantly after each label scan. My device is used by drivers to scan cases with barcodes, after every scan I do the following:

MutableDocument manifestroute = new MutableDocument("manifestroute");
                manifestroute.setString("RECEIVED_BAK_FILENAME", jsonModel.base.Manifest.Route.RECEIVED_BAK_FILENAME);
                manifestroute.setString("RETURN_XML_PATH", jsonModel.base.Manifest.Route.RETURN_XML_PATH);
                manifestroute.setBoolean("RETURN_XML_SUBMITTED", jsonModel.base.Manifest.Route.RETURN_XML_SUBMITTED);
                manifestroute.setString("DATE", jsonModel.base.Manifest.Route.DATE);
                manifestroute.setString("DC", jsonModel.base.Manifest.Route.DC);
                manifestroute.setString("INFOTEXT", jsonModel.base.Manifest.Route.INFOTEXT);
                manifestroute.setString("ROUTENO", jsonModel.base.Manifest.Route.ROUTENO);

            String customMappedInvoicesJson = h.ObjectToJson(jsonModel.base.Manifest.Route.CUSTOM_MAPPEDINVOICES, new TypeReference<Map<String, List<WrapperProduct>>>(){});
            Blob customMappedInvoicesBlob = new Blob("application/json", customMappedInvoicesJson.getBytes(Data.UTF8_CHARSET));
            manifestroute.setBlob("CUSTOM_MAPPEDINVOICES", customMappedInvoicesBlob);

            String deliveryListJson = h.ObjectToJson(jsonModel.base.Manifest.Route.Delivery, new TypeReference<List<WrapperProduct>>(){});
            Blob deliveryListBlob = new Blob("application/json", deliveryListJson.getBytes(Data.UTF8_CHARSET));
            manifestroute.setBlob("DELIVERY", deliveryListBlob);

            MutableDocument driver = new MutableDocument("driver");
            String driverJson = h.ObjectToJson(jsonModel.CurrentDriver, cb.cheneytracker.models.userfile.Driver.class);
            Blob driverBlob = new Blob("application/json", driverJson.getBytes(Data.UTF8_CHARSET));
            driver.setBlob("driver", driverBlob);

            MutableDocument mobiusroute = new MutableDocument("mobiusroute");
            String mobiusRouteJson = h.ObjectToJson(jsonModel.base.CadecData.MOBIUSROUTES, cb.cheneytracker.models.cadecdata.MOBIUSROUTES.class);
            Blob mobiusRouteBlob = new Blob("application/json", mobiusRouteJson.getBytes(Data.UTF8_CHARSET));
            mobiusroute.setBlob("mobiusroute", mobiusRouteBlob);

            MutableDocument deliverycancelled = new MutableDocument("deliverycancelled");
            String deliveryCancelledJson = h.ObjectToJson(jsonModel.CancelledMap, new TypeReference<Map<String, ManifestDelivery>>(){});
            Blob deliveryCancelledBlob = new Blob("application/json",deliveryCancelledJson.getBytes(Data.UTF8_CHARSET));
            deliverycancelled.setBlob("deliverycancelled", deliveryCancelledBlob);

            MutableDocument invoicecancelled = new MutableDocument("invoicecancelled");
            String invoiceCancelledJson = h.ObjectToJson(jsonModel.CancelledMapInvoices, new TypeReference<Map<String, List<ManifestInvoice>>>(){});
            Blob invoiceCancelledBlob = new Blob("application/json",invoiceCancelledJson.getBytes(Data.UTF8_CHARSET));
            invoicecancelled.setBlob("invoicecancelled", invoiceCancelledBlob);

            MutableDocument reportprinting = new MutableDocument("reportprinting");
            String reportPrintingJson = h.ObjectToJson(jsonModel.DoneScreenReportsMapping, new TypeReference<Map<String, List<DoneScreenReportParentModel>>>(){});
            Blob reportPrintingBlob = new Blob("application/json",reportPrintingJson.getBytes(Data.UTF8_CHARSET));
            reportprinting.setBlob("reportprinting", reportPrintingBlob);

            MutableDocument revisions = new MutableDocument("revisions");
            String revisionHashJson = h.ObjectToJson(jsonModel.RevisionHashModels, new TypeReference<Map<String, RevisionHashModel>>(){});
            Blob revisionHashBlob = new Blob("application/json",revisionHashJson.getBytes(Data.UTF8_CHARSET));
            revisions.setBlob("revisions", revisionHashBlob);

            MutableDocument poa = new MutableDocument("poa");
            String poaJson = h.ObjectToJson(jsonModel.PoaPaymentCollection, new TypeReference<List<PayPOAModelMapping>>(){});
            Blob poaJsonBlob = new Blob("application/json",poaJson.getBytes(Data.UTF8_CHARSET));
            poa.setBlob("poa", poaJsonBlob);

            MutableDocument outofstock = new MutableDocument("outofstock");
            String outOfStockJson = h.ObjectToJson(jsonModel.OutOfStockDelivery, new TypeReference<List<ManifestDelivery>>(){});
            Blob outOfStockJsonBlob = new Blob("application/json", outOfStockJson.getBytes(Data.UTF8_CHARSET));
            outofstock.setBlob("outofstock", outOfStockJsonBlob);

            MutableDocument productspec = new MutableDocument("productspec");
            String productSpecJson = h.ObjectToJson(jsonModel.base.Manifest.ProductSpec, new TypeReference<List<ProductSpec>>(){});
            Blob productSpecBlob = new Blob("application/json", productSpecJson.getBytes(Data.UTF8_CHARSET));
            productspec.setBlob("productspec", productSpecBlob);

            Data.database.save(manifestroute);
            Data.database.save(mobiusroute);
            Data.database.save(driver);
            Data.database.save(deliverycancelled);
            Data.database.save(invoicecancelled);
            Data.database.save(reportprinting);
            Data.database.save(revisions);
            Data.database.save(poa);
            Data.database.save(outofstock);
            Data.database.save(productspec);

I notice the exception posted above happens after 70+ items scanned, I scan quickly one after the other. I ran android profiler and memory is properly clearing and cpu usage is low even when scanning constantly.

Note that I'm able to still read the database after the above errors take place, it just appears to be missing one of the models that i'm saving. I'm still debugging to see if i can gather more information.

Not sure if this is a bug on your end? Thanks!

UPDATE

I did a bit more testing on my end. It happens when attempting to save the outofstock json blob after x amount of saves. This particular line Data.database.save(outofstock);.

outofstock is an empty array [].

Here is where exception took place:

[bmeike]

Tracking this in https://issues.couchbase.com/browse/CBL-346

[bmeike]

Are you able to drive this in a debugger? I believe that the error takes place in the method C4Document.update near line 160. I suspect that _handle is 0 because the Document has been freed. If you can get confirmation, that would be fantastic.

[ddelapaz]

Are you able to drive this in a debugger? I believe that the error takes place in the method C4Document.update near line 160. I suspect that _handle is 0 because the Document has been freed. If you can get confirmation, that would be fantastic.

Perfect, let me check into it and will let you know in a few. Thanks!

[bmeike]

@ddelapaz : Also, if you have code that you can give me, that will repro the problem It would be a big help.

[bmeike]

@ddelapaz : I have code that attempts to repro this bug. It fails to do so in 1000 iterations running the code you show above. Happy to share it with you, if it will help you to drive the bug. I need to be able to reproduce this to do anything about it.

[ddelapaz]

Sorry for the delay, It would be great if you share that code you have, I can pair it with real data i have on my end to try and help you reproduce this. I'm also going to see if i can give you the most recent DB file i have that causes the problem.

Thank you

[bmeike]

Code is here: https://github.com/bmeike/CBL346

[ddelapaz]

I'm working on a db file you can use on your end , as well as some code from my end to try to isolate the problem better. I went over the code you have there; shouldn't

Blob outOfStockJsonBlob = new Blob("application/json", new byte[0]); be Blob outOfStockJsonBlob = new Blob("application/json", outOfStockJson.getBytes(Data.UTF8_CHARSET)); .

In the live example I have, this is where it crashes, on the OutOfStock section after x amount of tries, OutOfStock is always an empty array.

In regards to what you previously asked:

I believe that the error takes place in the method C4Document.update near line 160. I suspect that _handle is 0 because the Document has been freed. If you can get confirmation, that would be fantastic.

I stepped thru debugger and this is the data i gathered right before it crashed. I couldn't step in any further: static native long create2(long db, String docID, long body, int flags) throws LiteCoreException;

[bmeike]

A couple of things. 1) the issue that you cite immediately above is different than the one that you originally posted. The original issue is a failure in update2: at com.couchbase.litecore.C4Document.update2(Native Method) ~[na:0.0]

2) You said:

t happens when attempting to save the outofstock [...] outofstock is an empty array [].

We haven't seen this happen elsewhere. If you can reproduce it in either the framework that I sent over, or some other code that you can share, we'll be able to fix it for you.

[ddelapaz]

I was able to reproduce: https://github.com/ddelapaz/CBL346_BugTest

What I found weird is that the error doesn't take place in android emulators OS v6, 7 or 8.. Works fine there. Also works fine on physical android devices v7, 8 but not on 6...

On a physical device with OS v 6.0.1 I get the error above 999 iterations.

Using the code I posted, this is after 999 iterations:

09-17 15:23:22.158 11130-11318/cb.dbbugreproduce D/DBHelper: DBHelper: OK, Saving 'outofstock' 09-17 15:23:22.204 11130-11318/cb.dbbugreproduce E/CouchbaseLite/DATABASE: Assertion failed: docID && revID && sequence > _lastSequence (/home/couchbase/jenkins/workspace/couchbase-lite-android-edition-build/couchbase-lite-android-ee/libs/couchbase-lite-android/libs/couchbase-lite-core/LiteCore/Database/SequenceTracker.cc:113, in documentChanged)/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so litecore::SequenceTracker::documentChanged(fleece::alloc_slice const&, fleece::alloc_slice const&, unsigned long, unsigned long)/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so c4Internal::Database::documentSaved(c4Internal::Document*)/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so c4Internal::TreeDocument::save(unsigned int)/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so c4Internal::TreeDocument::saveNewRev(C4DocPutRequest const&, litecore::Rev const*, bool)/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so c4Internal::TreeDocument::putNewRevision(C4DocPutRequest const&)/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so c4doc_put/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so c4doc_update/data/app/cb.dbbugreproduce-2/lib/arm64/libLiteCoreJNI.so Java_com_couchbase_litecore_C4Document_update2/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec/data/app/cb.dbbugreproduce-2/oat/arm64/base.odex oatexec??? /system/lib64/libart.so art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)/system/lib64/libart.so artInterpreterToCompiledCodeBridge/system/lib64/libart.so bool art::interpreter::DoCall<false, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*)??? /system/lib64/libart.so art::JValueart::interprete

Tomorrow I'm going to test it in a different hardware running 6.0.1 to see if its hardware related.

Hope this helps you.

[bmeike]

Nice work, @ddelapaz ! Will attempt to chase this down later this week. Many thanks for the help!

[ddelapaz]

Nice work, @ddelapaz ! Will attempt to chase this down later this week. Many thanks for the help!

Not a problem, thank you for being on top of it! I will update once if i get more details tomorrow. Have a good one.

[ddelapaz]

Quick update, the physical device that previously had this error running OSv6.0.1 now works properly after flashing OSv7.1.2. I'm trying to find a device that is not the Zebra TC75X to see if that makes a difference.

So currently the bug is only reproducible on a v6.0.1 Physical Device.

[bmeike]

Thanks for the info. That's weird!

[ddelapaz]

I was able to further confirm this issue on another Zebra TC75X physical device running OS 6.0.1.

I can't get my hands on a normal 6.0.1 device that is not Zebra, would you guys be able to test this on your end?

[bmeike]

I'm afraid that the only devices I have are a 5.1.1 and a 9. Emulator?

[ddelapaz]

I'm starting to think this is Zebra firmware specific issue and not related to the general Android releases. It doesn't happen on your 5.1.1?

[bmeike]

@ddelapaz : No... I have not been able to reproduce it anywhere...

[bmeike]

@ddelapaz : Do you have suggestions for what to do with this? I have yet to reproduce it anywhere and it seems hardware specific. If it is evidence of a bug others just haven't driven yet, I'd like to find it and fix it. At this point, though, there's just nothing to go on...

[ddelapaz]

I've been trying to get my hands on a device running OSv6.0.1 which is not Zebra specific but no luck. I want to find one of those devices to ensure that this is not just Zebra platform related.

[bmeike]

I'm going to close this. It's not that I don't believe that there is a bug, it is that I can't do much about it without more information.

If you come up with anything else, by all means, re-open this ticket or refer to it in a new one.

[benjaminglatzeder]

I'm just going through the release notes of Couchbase Lite 2.7 and saw this report. I have a Nexus 5 running Android 6.0.1 - which @ddelapaz was looking for. I grabbed https://github.com/ddelapaz/CBL346_BugTest, ran the loop 10,000 times and there were no issues/crashes/exception logs.

[bmeike]

Thank you @benjaminglatzeder !