search

EsotericSoftware / kryo

Java binary serialization and cloning: fast, efficient, automatic
BSD 3-Clause "New" or "Revised" License
6.19k stars 823 forks source link

Problem of migration ArrayList from 4.0.2 to 5.0.1 #795

Closed mspnr closed 3 years ago

mspnr commented 3 years ago

Description

On migration a project from Kryo 4.0.2 to 5.0.1 I've encountered a problem:

Analysis

After some parallel debugging I've ended up in the class CollectionSerializer.java

In the buffer I have the following data:

130 -> 92 -- start position and the length of the list
131 -> 13 -- ID of my class
132 -> 64 -- something, which becomes unregistered class 62

All generic types and serializers are null in both classes, so they are jumped over all ifs.

Screenshots

See the screenshots with code comparison below:

4.0.2

nlxrvA8txi

5.0.1

Pd2F5j1aqZ

Assumptions and questions

theigl commented 3 years ago

@applikationsprogramvara:

Could you take a look and recommend how to perform the migration? Or should one assume the new version as a totally different software?

Kryo 5 is not serialization compatible with Kryo 4 in most cases. If you have persistent data that you cannot recreate and need to migrate, you have a couple of options as outlined here: https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5

Kryo 5 offers a separate artifact that you can use in your application along with Kryo 4. So you could read old data with Kryo 4 and persist it with Kryo 5.

mspnr commented 3 years ago

Thank you for your feedback! Missing backward compatibility is for sure confusing.

As you recommended and as an experiment I've managed to import the both Kryo 4 and Kryo 5 from jars simultaneously into one program. It looks like with some effort and by duplicating all serializers for both versions, reading of the old data can work. E.g. I can read the old data as Kryo 4 and write them back as Kryo 5 with a special mark, so I can read them as Kryo 5 the next time. The implementation though becomes somehow tangled.

The main question is: What advantages / improvements has Kryo 5 over Kryo 4? Is is so much faster, less buggy or just simply better? Is it worth it to pursue the latest version instead of sticking with the old one?

The common sense tells me, that the latest versions are usually better, because they have less bugs, better organized, have more functions etc. But the fact, that I never had any issue with Kryo 4 together with this compatibility problem is stopping me to do it right away.

theigl commented 3 years ago

What advantages / improvements has Kryo 5 over Kryo 4? Is is so much faster, less buggy or just simply better? Is it worth it to pursue the latest version instead of sticking with the old one?

To quote the migration guide:

Also consider that if an older version is working fine, it may not be worth the pain to update.

Kryo 5 can be slightly faster for some scenarios. It is quite stable already, but since some core components were practically rewritten, I expect some minor bugs to surface in the coming months. It already supports JDK11 immutable lists and will soon add support for JDK15 records. If Kryo 4 currently works fine for you and you do not need support for new JDK data types, you can definitely stay with Kryo 4 for now.

mspnr commented 3 years ago

After some wandering I've managed to let the both versions of Kryo work together. And somehow it appeared to be not very tangled.

Although one of the plaform I target is Android with relatively low SDK, so on building I get a message:

MethodHandle.invoke and MethodHandle.invokeExact are only supported starting with Android O (--min-api 26)

Which makes using Kryo 5 impossible to used on SDK under 26. So yes, I will stick with Kryo 4 for now.

A couple off-topic ideas:

  1. What I was missing for Kryo is tutorials and examples. It would be great if you can add some into the Wiki. The Readme is good, but the whole tool is so complicated, so it would be great to create a simple example for every paragraph or serializer. As I read that, I thought, it looks like what I need. At this place I would expect a simple example, that I can copy-paste and play with it, but it is only partial or missing. The only comprehensive tutorial for Kryo I found is on baeldung.com, but it is only for basics.

  2. Regarding simple copy-paste examples. Here are a couple of notes regarding migration. That's what I was missing in my researches. You can use them as a starting point for the wiki-page regarding migration:

How to use Kryo4 and Kryo 5 together

Step 1: Update dependencies in build.gradle

Option 1: Using Maven

dependencies {
  api 'com.esotericsoftware:kryo:4.0.2'
  api 'com.esotericsoftware.kryo:kryo5:5.0.1'
}

Option 2: Using jars

Extract kryo-4.0.2.jar from https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.2 and place to libs folder Extract kryo5-5.0.1.jar from https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-5.0.1 and place to libs folder

dependencies {
  api "org.objenesis:objenesis:3.1"
  api "com.esotericsoftware:minlog:1.3.1"
  api "com.esotericsoftware:reflectasm:1.11.9"
  api files("libs/kryo-4.0.2.jar")
  api files("libs/kryo5-5.0.1.jar")
}

Step 2: Separate reading Kryo 4 data and reading/writing Kryo 5 data:

Use for Kryo 4 instances:

import com.esotericsoftware.kryo.Kryo;
import com.esotericsoftware.kryo.io.Input;

Use for Kryo 5 instances:

import com.esotericsoftware.kryo.kryo5.Kryo;
import com.esotericsoftware.kryo.kryo5.io.Input;
import com.esotericsoftware.kryo.kryo5.io.Output;

General mnemonic on reading is the following:

if (kryo4Version)
  return kryo4.readObject(input4, ArrayList.class);
else
  return kryo5.readObject(input5, ArrayList.class);

For writing it makes sense to use Kryo 5 only:

kryo5.writeObject(output5, list);

In this way you can migrate flawlessly from Kryo 4 to Kryo 5.

The topic can be closed. Thank you for your feedback!

theigl commented 3 years ago

@applikationsprogramvara: Thanks a lot for the feedback! Is it alright if I add the documentation you prepared to the migration guide?

Which makes using Kryo 5 impossible to used on SDK under 26.

You can use Kryo 5 with Android API < 26 if you override the Objenesis dependency version to 2.6 (see #691).

I'll add a section about Android to the readme as well.

mspnr commented 3 years ago

Documentation

Is it alright if I add the documentation you prepared to the migration guide?

Yes! Please use the documentation in the migration guide!

Android

This is a very good point you mentioned regarding Android. But it is still not working correctly without tweaking.

What is working: regular Kryo 5 version with overwritten Objenesis to 2.6:

implementation ('com.esotericsoftware:kryo:5.0.1') {
  exclude group: "org.objenesis"
}
//noinspection GradleDependency
implementation 'org.objenesis:objenesis:2.6'

What is not working: Kryo 5 library version:

implementation ('com.esotericsoftware.kryo:kryo5:5.0.1') {
  exclude group: "org.objenesis"
}
//noinspection GradleDependency
implementation 'org.objenesis:objenesis:2.6'

Rebuild (task mergeExtDexDebug) ends up with error: AGPBI: {"kind":"error","text":"com.android.tools.r8.a: MethodHandle.invoke and MethodHandle.invokeExact are only supported starting with Android O (--min-api 26)","sources":[{"file":"/home/abc/.gradle/caches/transforms-2/files-2.1/8bcb0db127fbb3f018615164ee628354/jetified-kryo5-5.0.1.jar"}],"tool":"D8"}

What is working with some tweaking

Could please tell you opinion regarding the approach and give some advice how to let it work with maven version?

theigl commented 3 years ago

Yes! Please use the documentation in the migration guide!

Thanks you!

What is not working: Kryo 5 library version

You are right. The library version includes all shaded dependencies so exclusion via Maven/Gradle does not work.

I'm not sure how we should deal with this issue. One option would be to revert back to Objenesis 2.6 as has been suggested in #691. I'm reluctant to make this change though because newer Objenesis versions contain some important fixes for JDK11.

Could please tell you opinion regarding the approach and give some advice how to let it work with maven version?

Your approach for regular Kryo is exactly right. I'll add this to the documentation for Android.

mspnr commented 3 years ago

Thank you for your feedback! I am happy, that I am on the right track.

What still confuses me, is that usually the libraries are ready to work "out of the box". You just write down the dependency to gradle, copy-paste the example and it is working.

Currently in Kryo 5 is the whole platform is not working without tweaking. I would suggest to to publish "Android-specific"-jar or Maven-library in the next release, so the developers would not be confused.

Yes, I did the hacking and let the basic function work, but it does not give me any confidence that it will not break in some other scenario, or will stop working in the future.

Anyway thank you for your support. I will try to integrate Kryo 5 into the production version hoping, that everything will work and will be improved in future.

theigl commented 3 years ago

Thanks again for you constructive feedback @applikationsprogramvara!

Currently in Kryo 5 is the whole platform is not working without tweaking. I would suggest to to publish "Android-specific"-jar or Maven-library in the next release, so the developers would not be confused.

I agree that the situation for Android is less than optimal at the moment. However, #691 has been open for a long time and hasn't received a lot of 👍 or comments. So my guess was that this isn't an important problem to address.

We could publish Android-specific artifacts but we would need to do that for normal Kryo as well as the library version. Effectively doubling the current number of artifacts and complicating the build. We would also need a basic test infrastructure for Android so we do not break it accidentally in the future.

I think time would better be spent on resolving https://github.com/easymock/objenesis/issues/79. My understanding of the Android VM is very limited, but it seems that method handles are the main issue. If so, Objenesis could be made to work by adding a special case for Android to https://github.com/easymock/objenesis/blob/master/main/src/main/java/org/objenesis/instantiator/util/DefineClassHelper.java that uses sun.misc.Unsafe directly as done in version 2.6.

theigl commented 3 years ago

I added your examples to the migration guide and the Readme. 🎉

theigl commented 3 years ago

Closing this issue for now. Thanks again @applikationsprogramvara! Please open another ticket if you have further issues or questions.

theigl commented 3 years ago

@applikationsprogramvara: Objenesis 3.2 has recently been released and is supposed to resolve the issue with Android <26. Could you check if 3.2 solves the problems you were facing? If so, I'll upgrade the dependency in Kryo.

mspnr commented 3 years ago

@theigl sorry for the late answer. As I understood you already updated Objenesis to version 3.2 in kryo-5.1.0.

Currently I replaced all the messy lines and extra file mentioned above with a single line:

implementation ('com.esotericsoftware.kryo:kryo5:5.1.1')

All working and compiling correctly on Android. Thank you!

I would suggest to simplify On Android section in README.