search

pilgr / Paper

Paper is a fast NoSQL-like storage for Java/Kotlin objects on Android with automatic schema migration support.
Apache License 2.0
2.35k stars 234 forks source link

KryoException: Buffer underflow #4

Open pilgr opened 9 years ago

pilgr commented 9 years ago

Very rarely crash happening in AppDialer app using Paper, affects only 0,05% sessions.

Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
a (io.paperdb.PaperTable)
       at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
       at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)
       at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:127)
       at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:667)
       at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:557)
       at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:686)
       at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:204)
       at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:128)
       at io.paperdb.Paper.get(Paper.java:52)
       at name.pilgr.appdialer.Storage.getHistory(Storage.java:54)
       at name.pilgr.appdialer.search.StatsManager.init(StatsManager.java:18)
       at name.pilgr.appdialer.Engine.setFire(Engine.java:197)
       at name.pilgr.appdialer.Engine.doStartUp(Engine.java:235)
       at name.pilgr.appdialer.Engine$1.run(Engine.java:70)
       at name.pilgr.appdialer.util.CrashableTask.run(CrashableTask.java:18)
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)
       at java.util.concurrent.FutureTask.run(FutureTask.java:237)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
       at java.lang.Thread.run(Thread.java:841)
pilgr commented 9 years ago

Looks like happens only once per affected user. All further data read attempts are successful.

raviteja06 commented 8 years ago

@pilgr Hi, I was using this library as an experiment in one of my projects. But I am having this issue every time I tried to retrieve an object for the first time. It is a specific object only.

pilgr commented 8 years ago

@raviteja06 Is it 100% reproducible in your case? Could you write a unit test to reproduce it?

raviteja06 commented 8 years ago

@pilgr I guess it's mostly because of different things I added in the development process. I cleared the cache and tried it, Never got that again.

martinbonnin commented 8 years ago

I got this too on 0.1% of the sessions. I have never been able to reproduce on one of my devices so far but it is a significant cause of crashes. Also most of these crashes happen at startup which is not good.

martinbonnin commented 8 years ago

Could it be linked to https://github.com/EsotericSoftware/kryo/issues/128 ? In all cases, can you put a section about Thread Safety in the README.md ? I understand it's ok to call Paper.book().read/write() from multiple threads, am I correct ?

pilgr commented 8 years ago

@martinbonnin yes, you can call read/write from different threads.

fingertricks commented 8 years ago

@pilgr I'm also seeing this crash on a client's app, which uses paperdb v1.1:

Fatal Exception: io.paperdb.PaperDbException: Couldn't read/deserialize file /data/user/0/com.justeat.app.uk/files/log/counter.pt for table counter
       at io.paperdb.DbStoragePlainFile.readTableFile(SourceFile:241)
       at io.paperdb.DbStoragePlainFile.select(SourceFile:130)
       at io.paperdb.Book.read(SourceFile:65)
...
Caused by com.esotericsoftware.kryo.KryoException: Buffer underflow.
Serialization trace:
mContent (io.paperdb.PaperTable)
       at com.esotericsoftware.kryo.io.Input.require(SourceFile:199)
       at com.esotericsoftware.kryo.io.Input.readVarInt(SourceFile:373)
       at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(SourceFile:127)
       at com.esotericsoftware.kryo.Kryo.readClass(SourceFile:667)
       at com.esotericsoftware.kryo.serializers.ObjectField.read(SourceFile:118)
       at com.esotericsoftware.kryo.serializers.FieldSerializer.read(SourceFile:551)
       at com.esotericsoftware.kryo.Kryo.readObject(SourceFile:686)
       at io.paperdb.DbStoragePlainFile.readTableFile(SourceFile:222)
       at io.paperdb.DbStoragePlainFile.select(SourceFile:130)
       at io.paperdb.Book.read(SourceFile:65)
...

We're also seeing it in a very small number of crashes per session, but according to Crashlytics the number of crashes is greater than the number of users impacted (each affected user has seen the crash 1.3x on average).

Crashlytics device insights indicate that the crash has only been seen on HTC and OnePlus devices running Android 6.0 or 6.0.1. Does that correspond with what you're seeing with AppDialer?

hwangjr commented 6 years ago

I face this error too:

device: Mi 2
Android version: 5.0.2 

+--- io.paperdb:paperdb:2.6
|    +--- com.esotericsoftware:kryo:4.0.1
|    |    +--- com.esotericsoftware:reflectasm:1.11.3
|    |    |    \--- org.ow2.asm:asm:5.0.4
|    |    +--- com.esotericsoftware:minlog:1.3.0
|    |    \--- org.objenesis:objenesis:2.5.1
|    \--- de.javakaffee:kryo-serializers:0.40
|         \--- com.esotericsoftware:kryo:4.0.0 -> 4.0.1 (*)
Process: com.angelia.wancai, PID: 26251

io.paperdb.PaperDbException: Couldn't read/deserialize file /data/data/com.angelia.wancai/files/io.paperdb/key_app_configuration.pt for table key_app_configuration

at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:298)

at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:158)

at io.paperdb.Book.read(Book.java:73)

at io.paperdb.Book.read(Book.java:58)

at com.angelia.wancai.MainApplication.config(MainApplication.java:98)

at com.angelia.wancai.utils.AppConfigurationUtils$2.onResponse(AppConfigurationUtils.java:74)

at okhttp3.RealCall$AsyncCall.execute(RealCall.java:135)

at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)

at java.lang.Thread.run(Thread.java:818)

Caused by: com.esotericsoftware.kryo.KryoException: Buffer underflow.

Serialization trace:

mContent (io.paperdb.PaperTable)

at com.esotericsoftware.kryo.io.Input.require(Input.java:199)

at com.esotericsoftware.kryo.io.Input.readVarInt(Input.java:373)

at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:127)

at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:693)

at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)

at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543)

at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:712)

at io.paperdb.DbStoragePlainFile.readContent(DbStoragePlainFile.java:307)

at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:291)

... 10 more
steven274 commented 6 years ago

I am facing this error when .pt file show 0 bytes

Manikandan92 commented 5 years ago

I have updated the latest version of 2.6. While crash occurring on the latest version also. Please check the below track.

Caused by io.paperdb.PaperDbException: Couldn't read/deserialize file /data/user/0/xxx.xx.xxxxxxxxx/files/mallow/TableName.pt for table TableName.pt at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:298) at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:158) at io.paperdb.Book.read(Book.java:73) at xxx.xx.xxxxxxxxx(PaperDB.java:110) at xxx.xx.xxxxxxxxx(MainActivity.java:382) at xxx.xx.xxxxxxxxx(MainActivity.java:347) at android.app.Activity.performCreate(Activity.java:6701) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2677) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2789) at android.app.ActivityThread.-wrap12(ActivityThread.java) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1527) at android.os.Handler.dispatchMessage(Handler.java:110) at android.os.Looper.loop(Looper.java:208) at android.app.ActivityThread.main(ActivityThread.java:6267) at java.lang.reflect.Method.invoke(Method.java) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1063) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:924)

Caused by com.esotericsoftware.kryo.KryoException: Buffer underflow. Serialization trace: journeyState (com.fleetrover.datamodel.TrackCurrentLocation) mContent (io.paperdb.PaperTable) at com.esotericsoftware.kryo.io.Input.require(Input.java:199) at com.esotericsoftware.kryo.io.Input.readAscii_slow(Input.java:616) at com.esotericsoftware.kryo.io.Input.readAscii(Input.java:594) at com.esotericsoftware.kryo.io.Input.readString(Input.java:472) at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:199) at com.esotericsoftware.kryo.serializers.DefaultSerializers$StringSerializer.read(DefaultSerializers.java:188) at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:789) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132) at com.esotericsoftware.kryo.serializers.CompatibleFieldSerializer.read(CompatibleFieldSerializer.java:145) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:734) at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:543) at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:712) at io.paperdb.DbStoragePlainFile.readContent(DbStoragePlainFile.java:307) at io.paperdb.DbStoragePlainFile.readTableFile(DbStoragePlainFile.java:291) at io.paperdb.DbStoragePlainFile.select(DbStoragePlainFile.java:158) at io.paperdb.Book.read(Book.java:73) at xxx.xx.xxxxxxxxx(PaperDB.java:110) at xxx.xx.xxxxxxxxx(MainActivity.java:382) at xxx.xx.xxxxxxxxx(MainActivity.java:347) at android.app.Activity.performCreate(Activity.java:6701) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2677) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2789) at android.app.ActivityThread.-wrap12(ActivityThread.java) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1527) at android.os.Handler.dispatchMessage(Handler.java:110) at android.os.Looper.loop(Looper.java:208) at android.app.ActivityThread.main(ActivityThread.java:6267) at java.lang.reflect.Method.invoke(Method.java) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1063) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:924)

vzts commented 5 years ago

same here as @Manikandan92

condesales commented 4 years ago

I can reproduce the problem 100% of the times. For me, it's happening because I moved the class I was saving to a different package.

Joseph82 commented 4 years ago

We went in production recently (3 days ago) and we just registered two episodes: both for the same user.

Still no idea about what can cause it?

brandonpage commented 3 years ago

This issue is still present in version 2.7.1. It is rare and not immediately reproducible like 2.6 but it still happens. Please catch the exception within Paper.

The crash occurs within our SDK at this read line.

com.esotericsoftware.kryo.KryoException: Buffer underflow. Serialization trace: mContent (io.paperdb.PaperTable)


com.esotericsoftware.kryo.io.Input.readVarInt Input.java:373
com.esotericsoftware.kryo.util.DefaultClassResolver.readClass DefaultClassResolver.java:127
com.esotericsoftware.kryo.Kryo.readClass Kryo.java:693
com.esotericsoftware.kryo.serializers.ObjectField.read ObjectField.java:118
com.esotericsoftware.kryo.serializers.FieldSerializer.read FieldSerializer.java:543
com.esotericsoftware.kryo.Kryo.readObject Kryo.java:712
io.paperdb.DbStoragePlainFile.readContent DbStoragePlainFile.java:332
io.paperdb.DbStoragePlainFile.readTableFile DbStoragePlainFile.java:316
io.paperdb.DbStoragePlainFile.select DbStoragePlainFile.java:166
io.paperdb.Book.read Book.java:73
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchEvent EventStoreManager.java:123
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchAllEvents EventStoreManager.java:147
com.salesforce.androidsdk.analytics.SalesforceAnalyticsManager.publishAllEvents SalesforceAnalyticsManager.java:279
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:605
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:591
android.os.AsyncTask$2.call AsyncTask.java:333
java.util.concurrent.FutureTask.run FutureTask.java:266
android.os.AsyncTask$SerialExecutor$1.run AsyncTask.java:245
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java:1167
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java:641
java.lang.Thread.run Thread.java:764

io.paperdb.DbStoragePlainFile.readTableFile DbStoragePlainFile.java:323
io.paperdb.DbStoragePlainFile.select DbStoragePlainFile.java:166
io.paperdb.Book.read Book.java:73
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchEvent EventStoreManager.java:123
com.salesforce.androidsdk.analytics.store.EventStoreManager.fetchAllEvents EventStoreManager.java:147
com.salesforce.androidsdk.analytics.SalesforceAnalyticsManager.publishAllEvents SalesforceAnalyticsManager.java:279
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:605
com.salesforce.mobile.analytics.ept.SalesforceAILTNEvent$PublishTask.doInBackground SalesforceAILTNEvent.java:591
android.os.AsyncTask$2.call AsyncTask.java:333
java.util.concurrent.FutureTask.run FutureTask.java:266
android.os.AsyncTask$SerialExecutor$1.run AsyncTask.java:245
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java:1167
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java:641
java.lang.Thread.run Thread.java:764
brandonpage commented 3 years ago

Hello @pilgr, do you have any intention of fixing or catching this issue within Paper?

pilgr commented 3 years ago

@brandonpage since this is incredibly rare and hardly to be reproduced, I'm not sure how to fix it. We may add some workarounds like second read attempts though. I'm wondering how often does it happen comparing to the total number of the user sessions?

rufuszhu commented 3 years ago

@pilgr This buffer underflow crash is also happening in our app as we are using SalesforceMobileSDK, which use version 2.7.1 of paper db. This issue only affected a few of our users but once it happens it will crash consistently (crash 200+ times for some of our users). Also because this crash is happening to those users consistently, the buffer underflow crash has pushed our crash per session rate to almost 4%

pilgr commented 3 years ago

@rufuszhu my understanding is if issue happens all the time for particular user, that might be caused by app update which used new package name for the class holding data. Please check out this message mentioned above.

I can reproduce the problem 100% of the times. For me, it's happening because I moved the class I was saving to a different package.

If the class has been stored with one package name and then on read the different class is expected (even if only package name has been changed) that may cause the issue.

For the same exception which happens very rarely I assume some other root cause takes place, that is still unknown.