search

thelastpickle / cassandra-reaper

Automated Repair Awesomeness for Apache Cassandra
http://cassandra-reaper.io/
Apache License 2.0
490 stars 218 forks source link

Can't remove a deleted cluster on UI #592

Open pi4alina opened 5 years ago

pi4alina commented 5 years ago

Project board link

I recently ran into this issue when a cluster on our AWS account has been terminated, but I can't delete it on the Reaper UI under "Clusters". ( The cluster got removed before any change was made on Reaper UI. ) I removed the repair schedule for that cluster, then I clicked "Forget cluster" button, but the cluster still shows. In the Reaper log, I see error about failing to connect to those no longer exists nodes.


java.lang.RuntimeException: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 10.0.0.5:7199```

┆Issue is synchronized with this [Jira Story](https://datastax.jira.com/browse/REAP-160) by [Unito](https://www.unito.io)
┆Issue Number: REAP-160
michaelsembwever commented 5 years ago

@pi4alina what version of Reaper are you using?

michaelsembwever commented 5 years ago

@adejanovski , i can't see what in the delete resource call would require a jmx call? https://github.com/thelastpickle/cassandra-reaper/blob/master/src/server/src/main/java/io/cassandrareaper/resources/ClusterResource.java#L338-L362

@pi4alina could you provide a stack trace to the exception?

FrancoisLe-Labs commented 4 years ago

I have a similar issue with version reaper 2.0.5. Here is the error stack : INFO [2020-07-27 17:51:59,629] [dw-75 - DELETE /cluster/providencecluster] i.c.r.ClusterResource - delete cluster providencecluster ERROR [2020-07-27 17:52:00,315] [ClusterResource] i.c.j.JmxConnectionFactory - Failed creating a new JMX connection to 10.0.30.69:7199 java.lang.RuntimeException: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 10.0.30.69:7199 at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:222) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at io.cassandrareaper.jmx.JmxConnectionFactory.connectImpl(JmxConnectionFactory.java:102) at io.cassandrareaper.jmx.JmxConnectionFactory.connectAny(JmxConnectionFactory.java:137) at io.cassandrareaper.jmx.ClusterFacade.connectImpl(ClusterFacade.java:809) at io.cassandrareaper.jmx.ClusterFacade.connect(ClusterFacade.java:793) at io.cassandrareaper.jmx.ClusterFacade.getNodesStatus(ClusterFacade.java:206) at io.cassandrareaper.resources.ClusterResource.lambda$getEndpointState$4(ClusterResource.java:369) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.cassandrareaper.ReaperException: Failure when establishing JMX connection to 10.0.30.69:7199 at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:264) at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:168) at io.cassandrareaper.jmx.JmxConnectionFactory$JmxConnectionProvider.apply(JmxConnectionFactory.java:213) ... 14 common frames omitted Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 10.0.30.69; nested exception is: java.net.NoRouteToHostException: No route to host (Host unreachable)] at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:206) at io.cassandrareaper.jmx.JmxProxyImpl.connectWithTimeout(JmxProxyImpl.java:282) at io.cassandrareaper.jmx.JmxProxyImpl.connect(JmxProxyImpl.java:221) ... 16 common frames omitted Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: Exception creating connection to: 10.0.30.69; nested exception is: java.net.NoRouteToHostException: No route to host (Host unreachable)] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at io.cassandrareaper.jmx.JmxProxyImpl.lambda$connectWithTimeout$0(JmxProxyImpl.java:281) ... 4 common frames omitted Caused by: javax.naming.CommunicationException: null at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) at javax.naming.InitialContext.lookup(InitialContext.java:417) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ... 6 common frames omitted Caused by: java.rmi.ConnectIOException: Exception creating connection to: 10.0.30.69; nested exception is: java.net.NoRouteToHostException: No route to host (Host unreachable) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:631) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:338) at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:112) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ... 11 common frames omitted Caused by: java.net.NoRouteToHostException: No route to host (Host unreachable) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.(Socket.java:434) at java.net.Socket.(Socket.java:211) at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ... 16 common frames omitted

adejanovski commented 2 years ago

Hi @pi4alina, are you still experiencing this problem?

vbatuev commented 1 year ago

got same issue, unable to forgot previously added cluster cluster "cluster" cannot be deleted, as it has repair runs cassandra-reaper 3.3.4 cassandra 4.1.2