search

EsotericSoftware / kryo

Java binary serialization and cloning: fast, efficient, automatic
BSD 3-Clause "New" or "Revised" License
6.19k stars 823 forks source link

nullpointer when deserialize concurrent tress #813

Closed Chuqiaoo closed 3 years ago

Chuqiaoo commented 3 years ago

I got nullPointerException when Kryo deserialization, serialize the data and write into a file, works no exception, but exception is found when deserialize the content from a file.

Kryo version: 5.0.4 Concurrent tress: 2.6.1 JDK: 11 Spring: 5 Test code:

    @Test
    public void read() {
        Input input = null;
        try {
            Kryo kryo = new Kryo();
            kryo.setRegistrationRequired(false);
            kryo.setReferences(true);
            kryo.setInstantiatorStrategy(new DefaultInstantiatorStrategy(new StdInstantiatorStrategy()));
            //The file is generated by kryo.writeClassAndObject 
            String fileName = "/path/test.bin";
            input = new Input(new FileInputStream(fileName));
            kryo.readClassAndObject(input);
        } catch ( RuntimeException| FileNotFoundException ex) {
            ex.printStackTrace();
        } finally {
            if (input != null) input.close();
        }
    }

Exception:

com.esotericsoftware.kryo.KryoException: java.lang.NullPointerException
Serialization trace:
outgoingEdgesAsList (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
array (java.util.concurrent.atomic.AtomicReferenceArray)
outgoingEdges (com.googlecode.concurrenttrees.radix.node.concrete.charsequence.CharSequenceNodeNonLeafNullValue)
root (com.googlecode.concurrenttrees.radix.ConcurrentRadixTree)
tree (org.reactome.server.analysis.core.model.IdentifiersMap)
entitiesMap (org.reactome.server.analysis.core.model.DataContainer)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:145)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:356)
    at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:299)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:122)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:861)
    at org.reactome.server.analysis.core.SerializeTest.read(SerializeTest.java:58)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
    at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
    at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
    at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
Caused by: java.lang.NullPointerException at 
com.googlecode.concurrenttrees.radix.node.util.AtomicReferenceArrayListAdapter.size(AtomicReferenceArrayListAdapter.java:45)
    at java.base/java.util.AbstractList.add(AbstractList.java:111)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:241)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:44)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:780)
    at com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:123)
    ... 100 more
theigl commented 3 years ago

@Chuqiaoo: Kryo is not a drop-in replacement for standard Java serialization. It supports almost all built-in JDK classes and can serialize most POJOs you throw at it, but it may need help with complex data structures.

It might be enough to write your own serializer for AtomicReferenceArrayListAdapter. It extends AbstractList so Kryo's default CollectionSerializer is used. It uses Objensis to instantiate the class and the backing array in AtomicReferenceArrayListAdapter is null after instantiation leading to your NPE.

Implement a custom serializer for this class that serializes the backing array and correctly instantiates the class with the array upon deserialization.

theigl commented 3 years ago

Alternatively, you should be able to force Kryo to use FieldSerializer for this class:

kryo.register(AtomicReferenceArrayListAdapter.class, new FieldSerializer<>(kryo, AtomicReferenceArrayListAdapter.class));
Chuqiaoo commented 3 years ago

Alternatively, you should be able to force Kryo to use FieldSerializer for this class:

kryo.register(AtomicReferenceArrayListAdapter.class, new FieldSerializer<>(kryo, AtomicReferenceArrayListAdapter.class));

it really helps a lot, thank you so much, even though I already set setRegistrationRequired to false, still have to register AtomicReferenceArrayListAdapter.class because it is not built-in JDK classes, is that correct?

theigl commented 3 years ago

even though I already set setRegistrationRequired to false, still have to register AtomicReferenceArrayListAdapter.class because it is not built-in JDK classes, is that correct?

No, you don't have to register it. The problem with AtomicReferenceArrayListAdapter is that Kryo's default registration for this class is CollectionSerializer (because it implements List). And CollectionSerializer does not know how to create a valid new instance of AtomicReferenceArrayListAdapter.

The code I suggested overrides Kryo's default registration and tells it to use FieldSerializer instead of CollectionSerializer. FieldSerializer uses reflection to read/write all fields of the class and that works in your case.

Chuqiaoo commented 3 years ago

even though I already set setRegistrationRequired to false, still have to register AtomicReferenceArrayListAdapter.class because it is not built-in JDK classes, is that correct?

No, you don't have to register it. The problem with AtomicReferenceArrayListAdapter is that Kryo's default registration for this class is CollectionSerializer (because it implements List). And CollectionSerializer does not know how to create a valid new instance of AtomicReferenceArrayListAdapter.

The code I suggested overrides Kryo's default registration and tells it to use FieldSerializer instead of CollectionSerializer. FieldSerializer uses reflection to read/write all fields of the class and that works in your case.

ok, I get your point, your suggestion is very helpful and thank you for your reply.