search

smartcat-labs / cassandra-diagnostics

Cassandra Node Diagnostics Tools
Apache License 2.0
51 stars 6 forks source link

No response when configuration is reloaded over http API #214

Closed milannister closed 7 years ago

milannister commented 7 years ago

When diagnostics configuration is reloaded over http API the server returns empty response. However, it does reload the configuration.

~$ curl -XPOST "http://127.0.0.1:8998/reload"
curl: (52) Empty reply from server

Also, exception is found in log file

ERROR [NanoHttpd Request Processor (#1)] 2017-03-23 13:39:54,981 MetricsCollector.java:102 - Cannot connect to jmx on 127.0.0.1:7199
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_121]
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_121]
        at io.smartcat.cassandra.diagnostics.module.metrics.MetricsCollector.connect(MetricsCollector.java:94) ~[cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.module.metrics.MetricsModule.<init>(MetricsModule.java:48) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) [na:1.8.0_121]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [na:1.8.0_121]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [na:1.8.0_121]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [na:1.8.0_121]
        at io.smartcat.cassandra.diagnostics.DiagnosticsProcessor.createModule(DiagnosticsProcessor.java:87) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.DiagnosticsProcessor.initModules(DiagnosticsProcessor.java:64) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.DiagnosticsProcessor.<init>(DiagnosticsProcessor.java:44) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.Diagnostics.reload(Diagnostics.java:165) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.api.DiagnosticsApiImpl.reload(DiagnosticsApiImpl.java:39) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.api.HttpHandler.respond(HttpHandler.java:55) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at io.smartcat.cassandra.diagnostics.api.HttpHandler.serve(HttpHandler.java:41) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at fi.iki.elonen.NanoHTTPD$HTTPSession.execute(NanoHTTPD.java:945) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at fi.iki.elonen.NanoHTTPD$ClientHandler.run(NanoHTTPD.java:192) [cassandra-diagnostics-core-1.3.9-SNPASHOT.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]
Caused by: javax.naming.ServiceUnavailableException: null
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136) ~[na:1.8.0_121]
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) ~[na:1.8.0_121]
        at javax.naming.InitialContext.lookup(InitialContext.java:417) ~[na:1.8.0_121]
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955) ~[na:1.8.0_121]
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922) ~[na:1.8.0_121]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ~[na:1.8.0_121]
        ... 17 common frames omitted
Caused by: java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is: 
        java.net.ConnectException: Connection refused (Connection refused)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) ~[na:1.8.0_121]
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) ~[na:1.8.0_121]
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) ~[na:1.8.0_121]
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) ~[na:1.8.0_121]
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) ~[na:1.8.0_121]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132) ~[na:1.8.0_121]
        ... 22 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_121]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_121]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_121]
        at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_121]
        at java.net.Socket.connect(Socket.java:538) ~[na:1.8.0_121]
        at java.net.Socket.<init>(Socket.java:434) ~[na:1.8.0_121]
        at java.net.Socket.<init>(Socket.java:211) ~[na:1.8.0_121]
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40) ~[na:1.8.0_121]
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148) ~[na:1.8.0_121]
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613) ~[na:1.8.0_121]
        ... 27 common frames omitted

This is produced on local three-node (127.0.0.1, 127.0.0.2, 127.0.0.3) cluster created using ccm tool.

Relevant part of cassandra-diagnostics.yml:

# controls if HTTP API is turned on. 'true' by default.
httpApiEnabled: true
# specifies the host/address part for listening TCP socket. '127.0.0.1' by default.
httpApiHost: 127.0.0.1
# specifies the port number for the listening TCP socket. '8998' by default.
httpApiPort: 8998
# if API authorization is enabled, API key must be provided through the 'Authorization' header
httpApiAuthEnabled: false
# API access key
httpApiKey: "diagnostics-api-key"

connector:
  jmxHost: 127.0.0.1
  jmxPort: 7100
nivancevic commented 7 years ago

What would be an appropriate/expected result?

milannister commented 7 years ago

At least status 2xx should be returned.

milannister commented 7 years ago

Confirmed that it works correctly on v1.3.8:

milan@SmartCat-Inspiron-5559:~$ curl -XPOST "http://127.0.0.1:8998/reload"
Configuration reloaded