ClassCastException in AbstractInvocationFuture

[dimejiogunyoye]

Describe the bug We observed this in our logs today. Not sure if this is a problem in Hazelcast and/or our use of it

java.util.concurrent.CompletionException: java.lang.ClassCastException: class com.hazelcast.spi.impl.AbstractInvocationFuture$WhenCompleteNode cannot be cast to class java.lang.Void (com.hazelcast.spi.impl.AbstractInvocationFuture$WhenCompleteNode is in unnamed module of loader 'app'; java.lang.Void is in module java.base of loader 'bootstrap')
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) [?:?]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) [?:?]
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:718) [?:?]
        at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:479) [?:?]
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
        at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
        at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) [?:?]
Caused by: java.lang.ClassCastException: class com.hazelcast.spi.impl.AbstractInvocationFuture$WhenCompleteNode cannot be cast to class java.lang.Void (com.hazelcast.spi.impl.AbstractInvocationFuture$WhenCompleteNode is in unnamed module of loader 'app'; java.lang.Void is in module java.base of loader 'bootstrap')
        at com.hazelcast.spi.impl.DelegatingCompletableFuture$DeserializingConsumer.accept(DelegatingCompletableFuture.java:494) ~[hazelcast-4.0.3.jar:4.0.3]
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?]
        ... 6 more

To Reproduce No clear reproduction steps, this is the first I've observed it

Additional context Hazelcast 4.0.3

Cluster size: 1

openjdk version "11.0.6" 2020-01-14 OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.6+10) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.6+10, mixed mode)

Operating System: NAME="Alpine Linux" ID=alpine VERSION_ID=3.11.5 PRETTY_NAME="Alpine Linux v3.11"

Maybe related: This node is running in a k8s cluster. Around the same time, we had another node, in the same k8s cluster, run into an existing issue I raised last week. Not sure if this is related in any way, but didn't seem like a coincidence.

[vbekiaris]

hi @dimejiogunyoye . You mention you are running hazelcast with openjdk 11. Can you share your complete java command? In particular, can you check that it includes all the required module exports & opens as described in Running in Modular Java ?

[dimejiogunyoye]

Thanks again for getting back to me @vbekiaris These are some of the relevant flags we have set:

--add-opens java.base/sun.nio.ch=ALL-UNNAMED\
 --add-opens java.base/java.util=ALL-UNNAMED\
 \
 --add-modules java.se\
 --add-exports java.base/jdk.internal.ref=ALL-UNNAMED\
 --add-opens java.base/java.lang=ALL-UNNAMED\
 --add-opens java.base/java.nio=ALL-UNNAMED\
 --add-opens java.base/sun.nio.ch=ALL-UNNAMED\
 --add-opens java.management/sun.management=ALL-UNNAMED\
 --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED\

[vbekiaris]

I would suggest you to upgrade to latest Hazelcast version (5.0.2). There have been several fixes, including #18140 which deals with issues in DelegatingCompletableFuture that may have to do with the behaviour you observe in this issue and #20542. If you don't want to upgrade to 5.0.2, then you can also try with latest patch release of 4.2 or 4.1, however there is no 4.0 patch release yet that includes #18140.

[dimejiogunyoye]

Interesting thanks for that. We've observed this on Hazelcast 4.2 as well - did your fix make it into that or would we have to upgrade to latest patch (4.2.5)?

[vbekiaris]

@dimejiogunyoye the fix I mentioned above was in 4.2 also (#18070), so probably this is another issue. Are you still seeing this? Can you provide any info about your application, how you use hazelcast, your configuration etc?