Repairs are failing with "no replicas found for token range"

Repair cycles are failing continuously with "no replicas found for token range" after processing some segments. In every cycle it is failing at different segments.

Cassandra version: 3.11.4 Reaper version: 2.2.4 Repair parallelism: PARALLEL Incremental repair: false Repair threads: 4

ERROR [2021-07-15 10:49:53,376] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.j.JmxProxy - null ERROR [2021-07-15 10:49:53,377] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - RepairRun FAILURE, scheduling retry io.cassandrareaper.ReaperException: null at io.cassandrareaper.jmx.JmxProxyImpl.getLiveNodes(JmxProxyImpl.java:905) at io.cassandrareaper.jmx.ClusterFacade.getLiveNodes(ClusterFacade.java:194) at io.cassandrareaper.jmx.ClusterFacade.getLiveNodes(ClusterFacade.java:180) at io.cassandrareaper.service.RepairRunner.updateClusterNodeList(RepairRunner.java:231) at io.cassandrareaper.service.RepairRunner.run(RepairRunner.java:195) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:117) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:38) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.UndeclaredThrowableException: null at com.sun.proxy.$Proxy81.getLiveNodes(Unknown Source) at io.cassandrareaper.jmx.JmxProxyImpl.getLiveNodes(JmxProxyImpl.java:902) ... 16 common frames omitted Caused by: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:254) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:164) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:903) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:273) ... 18 common frames omitted Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.DataInputStream.readByte(DataInputStream.java:265) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:240) ... 23 common frames omitted ERROR [2021-07-15 11:50:26,733] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.j.JmxProxy - null ERROR [2021-07-15 11:50:26,738] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.j.ClusterFacade - [tokenRangeToEndpoint] no replicas found for token range io.cassandrareaper.core.Segment@7d5e463d io.cassandrareaper.ReaperException: java.util.concurrent.ExecutionException: io.cassandrareaper.ReaperException at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMap(ClusterFacade.java:281) at io.cassandrareaper.jmx.ClusterFacade.tokenRangeToEndpoint(ClusterFacade.java:354) at io.cassandrareaper.service.RepairRunner.repairSegment(RepairRunner.java:473) at io.cassandrareaper.service.RepairRunner.startNextSegment(RepairRunner.java:313) at io.cassandrareaper.service.RepairRunner.run(RepairRunner.java:192) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:117) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:38) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: io.cassandrareaper.ReaperException at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:481) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3755) at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2306) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) at com.google.common.cache.LocalCache.get(LocalCache.java:4147) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMap(ClusterFacade.java:277) ... 16 common frames omitted Caused by: io.cassandrareaper.ReaperException: null at io.cassandrareaper.jmx.JmxProxyImpl.getRangeToEndpointMap(JmxProxyImpl.java:326) at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMapImpl(ClusterFacade.java:776) at io.cassandrareaper.jmx.ClusterFacade.lambda$getRangeToEndpointMap$0(ClusterFacade.java:279) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ... 20 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at com.sun.proxy.$Proxy81.getRangeToEndpointMap(Unknown Source) at io.cassandrareaper.jmx.JmxProxyImpl.getRangeToEndpointMap(JmxProxyImpl.java:323) ... 26 common frames omitted Caused by: java.rmi.UnmarshalException: error unmarshalling return; nested exception is: java.net.SocketException: Connection reset at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:197) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) ... 28 common frames omitted Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2781) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2797) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3552) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3344) at java.io.ObjectInputStream.readString(ObjectInputStream.java:2023) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1649) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at java.util.ArrayList.readObject(ArrayList.java:799) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at java.util.HashMap.readObject(HashMap.java:1412) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:327) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:177) ... 32 common frames omitted ERROR [2021-07-15 11:50:26,738] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.j.ClusterFacade - [tokenRangeToEndpoint] no replicas found for token range io.cassandrareaper.core.Segment@43e6e5a2 io.cassandrareaper.ReaperException: java.util.concurrent.ExecutionException: io.cassandrareaper.ReaperException at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMap(ClusterFacade.java:281) at io.cassandrareaper.jmx.ClusterFacade.tokenRangeToEndpoint(ClusterFacade.java:354) at io.cassandrareaper.service.RepairRunner.repairSegment(RepairRunner.java:473) at io.cassandrareaper.service.RepairRunner.startNextSegment(RepairRunner.java:313) at io.cassandrareaper.service.RepairRunner.run(RepairRunner.java:192) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:117) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:38) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: io.cassandrareaper.ReaperException at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:461) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2453) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2417) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) at com.google.common.cache.LocalCache.get(LocalCache.java:4147) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMap(ClusterFacade.java:277) ... 16 common frames omitted Caused by: io.cassandrareaper.ReaperException: null at io.cassandrareaper.jmx.JmxProxyImpl.getRangeToEndpointMap(JmxProxyImpl.java:326) at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMapImpl(ClusterFacade.java:776) at io.cassandrareaper.jmx.ClusterFacade.lambda$getRangeToEndpointMap$0(ClusterFacade.java:279) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) ... 21 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at com.sun.proxy.$Proxy81.getRangeToEndpointMap(Unknown Source) at io.cassandrareaper.jmx.JmxProxyImpl.getRangeToEndpointMap(JmxProxyImpl.java:323) ... 26 common frames omitted Caused by: java.rmi.UnmarshalException: error unmarshalling return; nested exception is: java.net.SocketException: Connection reset at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:197) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) ... 28 common frames omitted Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2781) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2797) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3552) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3344) at java.io.ObjectInputStream.readString(ObjectInputStream.java:2023) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1649) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at java.util.ArrayList.readObject(ArrayList.java:799) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at java.util.HashMap.readObject(HashMap.java:1412) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:327) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:177) ... 32 common frames omitted ERROR [2021-07-15 11:50:26,738] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.j.ClusterFacade - [tokenRangeToEndpoint] no replicas found for token range io.cassandrareaper.core.Segment@64f0b89b io.cassandrareaper.ReaperException: java.util.concurrent.ExecutionException: io.cassandrareaper.ReaperException at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMap(ClusterFacade.java:281) at io.cassandrareaper.jmx.ClusterFacade.tokenRangeToEndpoint(ClusterFacade.java:354) at io.cassandrareaper.service.RepairRunner.repairSegment(RepairRunner.java:473) at io.cassandrareaper.service.RepairRunner.startNextSegment(RepairRunner.java:313) at io.cassandrareaper.service.RepairRunner.run(RepairRunner.java:192) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:117) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:38) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at com.codahale.metrics.InstrumentedScheduledExecutorService$InstrumentedRunnable.run(InstrumentedScheduledExecutorService.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: io.cassandrareaper.ReaperException at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:502) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:481) at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:142) at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3755) at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2306) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) at com.google.common.cache.LocalCache.get(LocalCache.java:4147) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMap(ClusterFacade.java:277) ... 16 common frames omitted Caused by: io.cassandrareaper.ReaperException: null at io.cassandrareaper.jmx.JmxProxyImpl.getRangeToEndpointMap(JmxProxyImpl.java:326) at io.cassandrareaper.jmx.ClusterFacade.getRangeToEndpointMapImpl(ClusterFacade.java:776) at io.cassandrareaper.jmx.ClusterFacade.lambda$getRangeToEndpointMap$0(ClusterFacade.java:279) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) ... 20 common frames omitted Caused by: java.lang.reflect.UndeclaredThrowableException: null at com.sun.proxy.$Proxy81.getRangeToEndpointMap(Unknown Source) at io.cassandrareaper.jmx.JmxProxyImpl.getRangeToEndpointMap(JmxProxyImpl.java:323) ... 26 common frames omitted Caused by: java.rmi.UnmarshalException: error unmarshalling return; nested exception is: java.net.SocketException: Connection reset at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:197) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298) ... 28 common frames omitted Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2781) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2797) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3552) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3344) at java.io.ObjectInputStream.readString(ObjectInputStream.java:2023) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1649) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at java.util.ArrayList.readObject(ArrayList.java:799) at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at java.util.HashMap.readObject(HashMap.java:1412) at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1184) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2296) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461) at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:327) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:177) ... 32 common frames omitted WARN [2021-07-15 11:50:26,738] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - Segment #4044b1a9-e4fb-11eb-8b3f-9b713306ddf8 is faulty, no potential coordinators for range: io.cassandrareaper.core.Segment@64f0b89b WARN [2021-07-15 11:50:26,738] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - Segment #403b3bbf-e4fb-11eb-8b3f-9b713306ddf8 is faulty, no potential coordinators for range: io.cassandrareaper.core.Segment@7d5e463d WARN [2021-07-15 11:50:26,738] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - Segment #40164e71-e4fb-11eb-8b3f-9b713306ddf8 is faulty, no potential coordinators for range: io.cassandrareaper.core.Segment@43e6e5a2 INFO [2021-07-15 11:50:26,925] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - Repair amount done 3109.0 INFO [2021-07-15 11:50:26,925] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - Repair amount done 3109.0 INFO [2021-07-15 11:50:26,927] [prod:4008b9d0-e4fb-11eb-8b3f-9b713306ddf8] i.c.s.RepairRunner - Repair amount done 3109.0

thelastpickle / cassandra-reaper

Repairs are failing with "no replicas found for token range" #1103