ArrayIndexOutOfBoundsException in IdentityMap.clear()

[ghost]

From tcmarti...@yahoo.com on June 24, 2013 22:27:59

What steps will reproduce the problem?

1. Instantiate Kryo instance.
2. Instantiate fairly complex composite Java class (my project uses one that I can't share)
3. Create 100+ copies of the complex instance in a loop What is the expected output? What do you see instead? Expect all 100+ copies to be made, but instead, I'm getting an ArrayIndexOutOfBoundsException at some point. What version of the Kryo are you using? 2.19, 2.20, 2.21 Please provide any additional information below. This problem does NOT exhibit itself in 2.17 or lower. It pops up only in 2.19 and above. I recognize the problem is probably very specific to the nature of my complex, composite Java object that I'm cloning (copying) and therefore may be difficult for you guys to replicate. If time permits, I may try to troubleshoot this for you, however, I can clearly see something must have changed between 2.17 & 2.19 that affects IdentityMap.clear() to cause this. Perhaps you guys will be able to see the difference and arrive at a conclusion fairly easily. If not, let me know and I'll try to have a deeper look myself when I get an opportunity. In the meantime, I'll go just use 2.17. Thanks

Original issue: http://code.google.com/p/kryo/issues/detail?id=115

[ghost]

From tcmarti...@yahoo.com on June 24, 2013 13:30:58

Incidentally, I believe the exception is thrown around line 362 in version 2.19:

valueTable[i] = null;

[ghost]

From romixlev on July 23, 2013 06:30:41

Hi,

Could you provide a bit more details about this issue? Kryo is used to serialize thousands or even millions of rather complex objects and usually it works. So, there should be something specific to your test-case that makes it fail. To find out what it is we need more information.

For starters, a stack-trace would be nice to have. At least the Kryo related part of it. Which class throws the exception, where it is called from, etc?

A test-case that helps to reproduce a problem would be even better. But it looks like you have problems disclosing it, right?

-Leo

[ghost]

From ice...@gmail.com on July 29, 2013 05:18:23

I have the same issue, here is the stack trace maybe it can help : java.lang.ArrayIndexOutOfBoundsException at com.esotericsoftware.kryo.util.IdentityMap.clear(IdentityMap.java:361) at com.esotericsoftware.kryo.Kryo.reset(Kryo.java:809) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:835)

I don't know if this is linked to the problem but it seems that when I have 14 objects in one of my maps everything is ok, when it reaches 15 this exception is thrown (HashMap<Integer, Object>, jvm 1.5).

[ghost]

From romixlev on July 29, 2013 05:28:50

@-icewil: Thanks a lot for your report. Could you provide a small test-case to reproduce the problem? Or at least how the structure of your classes looks like and how you configure your kryo instance (e.g. refrences, registration of classes, etc)? It would help a lot.

BTW you used 2.21? Or current snapshot, i.e. 2.22-SNAPSHOT? You use Kryo instance by a single thread only? Or is it a multithreaded app which tries to share a kryo instance between threads?

Thanks, -Leo

[ghost]

From ice...@gmail.com on July 29, 2013 09:03:16

I use 2.21 (it's working normally with 2.16, I don't tried other versions between these two) and I use a new instance each time but the application is multithreaded (I also tried to put a synchronized(myObject) before creating the kryo instance but it doesn't change anything). I tried to reproduce the problem on a smaller scale but my objects are very complicated sadly...

(Note that once serialized/unserialized with the java standard way, the error is never thrown.)

[ghost]

From romixlev on July 29, 2013 09:13:56

OK. The important thing is that your Kryo instances are not simultaneously by multiple threads.

Could you test with 2.22-SNAPSHOT to see if this error is still present? Just replace the version in your POM by 2.22-SNAPSHOT.

[ghost]

From romixlev on July 29, 2013 09:48:46

I tried to reproduce the problem on a smaller scale but my objects are very complicated sadly... That's a pity. Because we need a hint how to reproduce a problem. Currently all Kryo unit tests pass, i.e. they do not trigger a problem. But we need to find a way to reproduce it.

Would it be possible for you to provide a test that only shows a structure of your classes and how you construct objects, so that we can reproduce a problem? Only the structure is important for Kryo. You can remove all the business logic and rename the classes and fields, if you don't want to disclose any secrets due to corporate policies.

-Leo

[ghost]

From ice...@gmail.com on July 30, 2013 01:39:07

I tried with 2.22-SNAPSHOT, the error is still present. I don't have enough time to do what you propose actually. I will use 2.16 for the moment and retry later when I will have more time.

[ghost]

From nathan.s...@gmail.com on August 05, 2013 05:23:27

If you can give us the keys when you have 14 objects and also the key for the 15th object you try to add, likely we can reproduce the crash.

[ghost]

From ice...@gmail.com on August 09, 2013 02:18:03

I finally got time for debugging the issue. I found out that the real error is not in IdentityMap.clear() but in IdentityMap.resize(), because of "finally" statement the original error is replaced. I had a try/catch in Kryo.reference() method, I got this error in addition to the other :

java.lang.OutOfMemoryError at com.esotericsoftware.kryo.util.IdentityMap.resize(IdentityMap.java:448)

So the OutOfMemoryError corrupted the IdentityMap object, the new array is not created but the capacity has been increased and this causes an error when clearing the map in "finally" statement in Kryo.copy().

Now, I do not understand why Kryo needs so much memory (the IdentityMap.size is around 500-800 objects and the IdentityMap.keyTable size is always 33554468 before trying to change is size to 67108902).

[ghost]

From romixlev on August 09, 2013 02:43:22

@-icewil: This is a very good hint! Thanks for looking into this.

But if you are at it anyway, could you really try to record just the keys which are in the map when you try to add the last one which leads to the crash, as Nate suggested? Then we likely can reproduce the crash.

I guess it would be even better if you could record the order in which keys were added/removed to/from identity map.

[ghost]

From romixlev on August 14, 2013 08:15:28

@-icewil, @-tcmartin24: Nate committed some fixes which may improve the situation. It would be nice, if you could try with the trunk version.

Btw, the kryo-2.22-SNAPSHOT is available in the sonatype snapshots repo: https://oss.sonatype.org/content/repositories/snapshots/ (https://oss.sonatype.org/content/repositories/snapshots/com/esotericsoftware/kryo/kryo/) -Leo

[ghost]

From ice...@gmail.com on August 16, 2013 09:57:23

I am on vacation until September, I had no time for working on this since my last comment. You will have to wait, sorry.

[ghost]

From ice...@gmail.com on September 03, 2013 02:17:25

I tried with the latest 2.22 snapshot, the memory error seems random depending on memory status. An example :

java.lang.OutOfMemoryError at com.esotericsoftware.kryo.util.IdentityMap.resize(IdentityMap.java:469) at com.esotericsoftware.kryo.util.IdentityMap.putStash(IdentityMap.java:255) at com.esotericsoftware.kryo.util.IdentityMap.push(IdentityMap.java:249) at com.esotericsoftware.kryo.util.IdentityMap.put(IdentityMap.java:144) at com.esotericsoftware.kryo.Kryo.reference(Kryo.java:833) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:878) at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(UnsafeCacheFields.java:297) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:153) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(UnsafeCacheFields.java:297) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:153) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(UnsafeCacheFields.java:297) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(UnsafeCacheFields.java:297) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(UnsafeCacheFields.java:297) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:153) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(UnsafeCacheFields.java:297) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877)

I noticed a few weeks ago that immutable objects are stored in the identity map with the key equals to the value. I have many immutable objects (String, Double, BigDecimal, Integer, ...) so I wonder if this might be causing the memory issue. Is it useful to store immutable objects in the identity map ?

[ghost]

From romixlev on September 03, 2013 02:28:35

@-icewil: Nice to hear from you again!

Thanks for checking the latest snapshot. I'd like to ask a few questions:

1) Your stack trace shows that you use Unsafe-based FieldSerializer. Could you also try with the ASM-based one? Simply call kryo.setAsmEnabled(true); immediately after this kryo instance is created, but before any other operations on it.

2) It seems you use Kryo's support for circular references and shared objects in the object graph. Is it the case? Do you really need it in your application? If not, you could try to disable it by using kryo.setReferences(false);

In any case, Kryo should work properly even without these proposed changes. Most likely you hit a very subtle bug in Kryo and we would really like to fix it. But we need a way to reproduce it somehow. May be you could try to provide a test-case? Or may be at least you could provide a set of keys and order of operations on the identity map that leads to the crash?

-Leo

[ghost]

From ice...@gmail.com on September 03, 2013 06:20:39

1) Only the stack changed with this setting :

java.lang.OutOfMemoryError at com.esotericsoftware.kryo.util.IdentityMap.resize(IdentityMap.java:469) at com.esotericsoftware.kryo.util.IdentityMap.putStash(IdentityMap.java:255) at com.esotericsoftware.kryo.util.IdentityMap.push(IdentityMap.java:249) at com.esotericsoftware.kryo.util.IdentityMap.put(IdentityMap.java:144) at com.esotericsoftware.kryo.Kryo.reference(Kryo.java:833) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:557) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:153) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.ObjectField.copy(ObjectField.java:140) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:153) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.ObjectField.copy(ObjectField.java:140) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.ObjectField.copy(ObjectField.java:140) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.ObjectField.copy(ObjectField.java:140) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:153) at com.esotericsoftware.kryo.serializers.MapSerializer.copy(MapSerializer.java:17) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877) at com.esotericsoftware.kryo.serializers.ObjectField.copy(ObjectField.java:140) at com.esotericsoftware.kryo.serializers.FieldSerializer.copy(FieldSerializer.java:562) at com.esotericsoftware.kryo.Kryo.copy(Kryo.java:877)

What is the purpose of this setting ?

2) I already tried this setting before, it does not change anything but I do not really need circular references or shared objects (I only need my objects to be different from the original ones).

I tried to produce manually a test case multiple times without success. My original object is initialized with many database accesses, use a hundred conditions to be initialized the same way and have many maps/intern classes/lists/properties to fill. I tried to serialize my objet in a file before the crash with the java default serialization but the object doesn't seems to produce the crash once unserialized in a separate test class. The identity map contains around 600 objets when the crash occurs and the last keys inserted are different each time, I am just unable (and not allowed) to provide a relevant set of keys. I have at least 200MB free memory available before the clone method call. I will do further testing the next week.

[ghost]

From romixlev on September 03, 2013 06:40:44

Regarding kryo.setAsmEnabled(true): The purpose of this setting is to use the ASM-based backend. This backend uses run-time bytecode generation. It is faster than Java-reflection based approach, but slower than Unsafe-based approach. But on some platforms like Android or securiry-constrained environments the Unsafe-based approach cannot be used due to platform limitations. In such situations, ASM-based backend is used. BTW, ASM was the only backend until Kryo's changes earlier this year introduced a new Unsafe-based implementation.

Overall, it is good that you get a consistent failure in all modes, because it really indicates that this is a generic bug in Kryo and it is not specific for a given backend or mode of operation.

Of course, it is very pity that you cannot provide us with a reproducible test-case yet. But let's see if we can find the problem anyways.

Let me continue with questions: 3) According to your traces, you try to copy/clone an object graph and hit the exception. If you serialize into a file and then deserialize instead, do you get the same problem?

4) Can you try to provide us with the info about how this identityMap resizes and stats about it? E.g. how many elements were in the map, what was the amount of reserved elements, etc when it performs a resize operation. I.e. it would be interesting to know the dynamics of growth for this map.

It would be also useful to know the rough amount of unique objects you insert into this map. In theory, the upper bound should be the amount of objects in your object graph that you try to serialize/clone.

5) In your comment #3 on this issue, you were hitting the problem already with 15 objects in the IdentityMap. Has it gone now for that test-case and you see another problem now? Or is that test-case still failing?

[ghost]

From romixlev on September 03, 2013 07:03:52

@-icewill:

I am just unable (and not allowed) to provide a relevant set of keys

Actually, we do not need the values of keys, most likely. We need just their hash-values as computed by hash functions inside the IdentityMap (because decisions inside this map are based on those hash-values mostly) and some virtual names for each key, e.g. key1,...,keyX, so that we can e.g. understand that same key is inserted twice, or a given key is deleted, etc, even though we don't know the value of this key.

And I'd say that hash values of keys won't disclose anything about the keys themselves or their semantics.

I guess stats like this could be produced if you'd build Kryo from sources and add some simple debug print statements into put, putResize, resize and may be some other methods that change the state of the map. At least it could be worth a try.

-Leo

[ghost]

From ice...@gmail.com on September 16, 2013 09:52:10

I think I finally found out what really happens. It seems to be linked to my IBM JVM. The keys in identity map can be differents but can have the same System.identityHashCode() value sometimes with this JVM (it seems managed differently with Oracle JVM). Here is an example :

import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map;

import com.esotericsoftware.kryo.Kryo;

public class Test {

public static void main(String[] args) throws Exception {

    Map<Integer, List<String>> map2 = new HashMap<Integer, List<String>>();

    List<String> biggerList = null;

    for (int i = 0; biggerList == null; i++) {
        String obj = new String("test"+i);

        List<String> samehashlist = map2.get(System.identityHashCode(obj));
        if (samehashlist != null) {
            samehashlist.add(obj);
            if(samehashlist.size() >= 100) {

                biggerList = samehashlist;

            }
        } else {
            samehashlist = new ArrayList<String>();
            samehashlist.add(obj);
            map2.put(System.identityHashCode(obj), samehashlist);
        }
    }
    map2.clear();
    map2 = null;

    System.gc();

    Runtime runtime = Runtime.getRuntime();
    double mb = 1024.*1024.;
    System.out.println("Memory used : " + ((runtime.totalMemory() - runtime.freeMemory())/mb) + "/" + (runtime.totalMemory()/mb) + " max:"+ (runtime.maxMemory()/mb)); 

    Kryo k = new Kryo();
    k.copy(biggerList);

}

}

With with kryo > 2.16 releases (I have not tested again with the snapshot release) :

This code will cause a java heap space exception on Oracle JVM (because all string identity hash codes seems differents) : java.lang.OutOfMemoryError: Java heap space at java.lang.StringBuilder.toString(Unknown Source)

But it will cause my ArrayIndexOutOfBoundsException with my IBM JVM on kryo.copy trying to copy a list of 100 different String with the same hash code.

I tried with latest IBM JVM, it seems to work as the Oracle one but I can not use it. My IBM JVM comes from IBM WebSphere 7 and I can not change it. I think that an option for using a different (but slower) indentity map implementation using "obj1 == obj2" equality could be enough for me to fix the issue (or an option to disable the map).

For the questions : 3) In practice, it seems to be random. 4) I noticed the map resized often when putting a String or Integer or an other object with a identity hash code already in the map index but with a different object (I guess it should not be the case anyway). 5) My map only impacted the amount of data (especially strings), I think the risk of collision between identity hash codes was more important when my map growed.

[ghost]

From romixlev on September 16, 2013 10:52:07

@-icewill: Thanks a lot for your feedback. I think now we are getting closer to the solution. As for the System.identityHashCode(), I've found this: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6321873 So, basically, we cannot rely on the fact that two different objects which are not equal always have different identityHashCode values. I guess we need to rewrite Kryo's identity maps accordingly by removing this assumption. @NathanSweet: Can you look into this?

Status: Accepted

[NathanSweet]

We don't rely on hashCodes being unique.

Object a = new Object() {
    public int hashCode () {
        return 1;
    }
};
Object b = new Object() {
    public int hashCode () {
        return 1;
    }
};
Object c = new Object() {
    public int hashCode () {
        return 1;
    }
};
Object d = new Object() {
    public int hashCode () {
        return 1;
    }
};
IdentityMap map = new IdentityMap();
map.put(a, null);
map.put(b, null);
map.put(c, null);
map.put(d, null);

The map uses 3 slots for each hashCode. When "d" is put in the map, the 3 slots are taken, so the map uses the cuckoo algorithm to push objects into their alternate slot and make room for the new object. This fails because all 4 objects want to use the same 3 slots. The map then stores the problem object in the stash, which is a small fixed size array. If the stash gets full, the map will resize, allocating more space for the stash.

Using objects with the same hashCode will result in poorer performance with any hash map implementation (java.util.HashMap will become a linked list, which statistically walks random memory). If a huge number of your objects have the same hashCode then Kryo's map will need to resize a lot to have a big enough stash. It uses Math.max(3, (int)Math.ceil(Math.log(capacity)) * 2) for the stash size. In that case I would say to fix the hashCodes of your objects, patch Kryo to not use IdentityMap, or patch IdentityMap to have a larger stash size.

[nicolashenry]

The issue is still here with latest snapshot.

@NathanSweet As far as I know Kryo IdentityMap do not use hashCode() method from implemented classes but from default Object.hashCode() through System.identityHashCode() method. So, I do not have any control on hash codes used by the JVM with Kryo and your example is not in relation with the issue :

Object a = new Object() { public int hashCode() { return 1; } }; System.out.println(System.identityHashCode(a)); // does not print 1

I will not patch Kryo just for our application, there will be too much maintenance. I would like, if possible, set the implementation class to the system class java.lang.IdentityHashMap which is maybe slower but seems not buggy with a list of 100 String with the same hash code.

IceWil

[NathanSweet]

My example showed what happened if System.identityHashCode returns the same code. It can be run with ObjectMap, which does use hashCode, to see it has the effect I described.

Use of IdentityMap is not configurable, which is why I suggested patching it.

[nicolashenry]

You suggested to fix the hashCodes of my objects but I can not fix the JVM, I just wanted to point that out. Patching it would mean that Kryo is not a reliable solution for us. I already spent too much time finding where the issue is and I do not have any time for hacking Kryo sadly. Do you have a plan to fix that or to add a workaround? Adding an option to fully disable the object references search could be enough as a workaround for us since we do not need it. Thanks for your time spent in this issue.

[NathanSweet]

I've made references optional for copying: 0a1c7e326c8b5ffae06ac4f6e03a7fec4aea6753 Copying will be much faster when copy references are disabled, just be sure not to have circular references (or objects in the graph multiple times).

[nicolashenry]

It seems to make latest Kryo usable in our context when copyReferences is disabled (we do not have circular references). I hope we will not need to keep references in the future. Thank you very much.

[ash211]

Did you ever figure out the root cause of the issue? It sounded like there was a hash code collision theory, but you ended up just disabling the buggy code path via copyReferences.

I'm seeing a very similar stacktrace when calling this library from Apache Spark:

java.lang.ArrayIndexOutOfBoundsException: 536870951
    at com.esotericsoftware.kryo.util.IdentityObjectIntMap.clear(IdentityObjectIntMap.java:345)
    at com.esotericsoftware.kryo.util.MapReferenceResolver.reset(MapReferenceResolver.java:47)
    at com.esotericsoftware.kryo.Kryo.reset(Kryo.java:804)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:570)
    at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:104)
    at org.apache.spark.storage.DiskBlockObjectWriter.write(BlockObjectWriter.scala:179)
    at org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:179)
    at org.apache.spark.util.collection.ExternalAppendOnlyMap.insert(ExternalAppendOnlyMap.scala:145)
    at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:157)
    at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$4.apply(CoGroupedRDD.scala:154)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

The largest change here is that it's with the IdentityObjectIntMap not the IdentityMap but the classes look like they implement the same algorithm.

Apart from disabling reference tracking entirely (possible for me in this situation but not universally) is there anything better I can do?

[NathanSweet]

I don't think it was ever clear how the exception is possible. I would guess multi threaded usage, which isn't valid. Otherwise you could replace the maps with JDK equivalents.

[ash211]

It's very possible there's a threading issue since Spark is highly multi-threaded, but the design is that each Spark thread gets its own Kryo instance for serialization so we "shouldn't" be multi-threaded usage of a Kryo instance.

Do you have any tips for detecting multi-threaded usage by chance?

[NathanSweet]

Nothing special, maybe an AtomicBoolean you set to true at the start and end of a method. Check it before setting to true, if true then another thread is in the method.

On Wed, Sep 3, 2014 at 6:22 PM, Andrew Ash notifications@github.com wrote:

It's very possible there's a threading issue since Spark is highly multi-threaded, but the design is that each Spark thread gets its own Kryo instance for serialization so we "shouldn't" be multi-threaded usage of a Kryo instance.

Do you have any tips for detecting multi-threaded usage by chance?

— Reply to this email directly or view it on GitHub https://github.com/EsotericSoftware/kryo/issues/115#issuecomment-54323163 .

[mateuszef]

came across same issue as mentioned above (https://github.com/EsotericSoftware/kryo/issues/115#issuecomment-54282567), when serializing data...

java.lang.ArrayIndexOutOfBoundsException: 536870951
        at com.esotericsoftware.kryo.util.IdentityObjectIntMap.clear(IdentityObjectIntMap.java:466)
        at com.esotericsoftware.kryo.util.MapReferenceResolver.reset(MapReferenceResolver.java:65)
        at com.esotericsoftware.kryo.Kryo.reset(Kryo.java:888)
        at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:559)

I am using kryo 4.0.0 I will let you know some details, if I can manage to collect the... Interesting that in content of exception same number shows up as in ApacheSpark library - 536870951

[xguo27]

Interestingly I see the exact same index number which throws exception.

java.lang.ArrayIndexOutOfBoundsException: 536870951
    at com.esotericsoftware.kryo.util.IdentityObjectIntMap.clear(IdentityObjectIntMap.java:382)
    at com.esotericsoftware.kryo.util.MapReferenceResolver.reset(MapReferenceResolver.java:65)
    at com.esotericsoftware.kryo.Kryo.reset(Kryo.java:865)
    at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:630)
    at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:241)
    at org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:134)

@mateuszef , how did you solve it?

[NathanSweet]

FWIW, all the Kryo maps have been updated recently in the master branch.

[Heltman]

I got this problem when use spark2.3.2: java.lang.ArrayIndexOutOfBoundsException: 536870951 at com.esotericsoftware.kryo.util.IdentityObjectIntMap.clear(IdentityObjectIntMap.java:382) at com.esotericsoftware.kryo.util.MapReferenceResolver.reset(MapReferenceResolver.java:65) at com.esotericsoftware.kryo.Kryo.reset(Kryo.java:865) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:630) at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:241) at org.apache.spark.serializer.SerializationStream.writeValue(Serializer.scala:134) at org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:241) at org.apache.spark.util.collection.ExternalAppendOnlyMap.org$apache$spark$util$collection$ExternalAppendOnlyMap$$spillMemoryIteratorToDisk(ExternalAppendOnlyMap.scala:233) at org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:185) at org.apache.spark.util.collection.ExternalAppendOnlyMap.spill(ExternalAppendOnlyMap.scala:55) at org.apache.spark.util.collection.Spillable.maybeSpill(Spillable.scala:97) at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:159) at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41) at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:90) at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:141) at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:137)