search

fabric8-services / keycloak

Open Source Identity and Access Management For Modern Applications and Services
http://www.keycloak.org
Other
1 stars 4 forks source link

Cluster node becomes useless #70

Closed hectorj2f closed 7 years ago

hectorj2f commented 7 years ago

In a cluster after some hammering, one node becomes useless cause each request returns a 500 error.

Solution: Destroy all the nodes of the cluster and re-create the cluster OR destroy the owner node and wait for a rebalance.

The exception that can be found in the node is:

[[0m^[[31m09:06:19,552 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-467) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from 10.1.16.6, see cause for remote stack trace
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:757)
        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$73(JGroupsTransport.java:612)
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
        at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
        at org.infinispan.remoting.transport.jgroups.RspListFuture.futureDone(RspListFuture.java:31)
        at org.jgroups.blocks.Request.checkCompletion(Request.java:169)
        at org.jgroups.blocks.GroupRequest.receiveResponse(GroupRequest.java:147)
        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:398)
        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:684)
        at org.jgroups.JChannel.up(JChannel.java:738)
        at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:119)
        at org.jgroups.stack.Protocol.up(Protocol.java:374)
        at org.jgroups.protocols.FORK.up(FORK.java:114)
        at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1045)
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
        at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
        at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:310)
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:285)
        at org.jgroups.protocols.Discovery.up(Discovery.java:295)
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
        at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.EOFException: Read past end of file
        at org.jboss.marshalling.SimpleDataInput.eofOnRead(SimpleDataInput.java:151)
        at org.jboss.marshalling.SimpleDataInput.readUnsignedByteDirect(SimpleDataInput.java:294)
        at org.jboss.marshalling.SimpleDataInput.readUnsignedByte(SimpleDataInput.java:249)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        at org.infinispan.commons.marshall.MarshallUtil.unmarshallMap(MarshallUtil.java:42)
        at org.infinispan.marshall.exts.MapExternalizer.readObject(MapExternalizer.java:83)
        at org.infinispan.marshall.exts.MapExternalizer.readObject(MapExternalizer.java:28)
        at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
        at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
        at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.readObject(ImmortalCacheValue.java:130)
        at org.infinispan.container.entries.ImmortalCacheValue$Externalizer.readObject(ImmortalCacheValue.java:122)
        at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
        at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
        at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:92)
        at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.readObject(SuccessfulResponse.java:77)
        at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:475)
        at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:234)
        at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:149)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
        at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:134)
        at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
        at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
hectorj2f commented 7 years ago

Closing it! We don't experience this behavior anymore.