search

alibaba / otter

阿里巴巴分布式数据库同步系统(解决中美异地机房)
Apache License 2.0
8.07k stars 2.5k forks source link

node_info.htm报错java.rmi.NoSuchObjectException: no such object in table #612

Open joooohnli opened 6 years ago

joooohnli commented 6 years ago

说明

部署了两台node，一台显示正常，一台异常

日志

Caused by:

java.rmi.NoSuchObjectException: no such object in table
    at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
    at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
    at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
    at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
    at com.alibaba.otter.manager.biz.remote.impl.NodeMBeanServiceImpl$1.apply(NodeMBeanServiceImpl.java:76)
    at com.alibaba.otter.manager.biz.remote.impl.NodeMBeanServiceImpl$1.apply(NodeMBeanServiceImpl.java:57)
    at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:356)
    at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:182)
    at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:151)
    at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:67)
    at com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:885)
    at com.alibaba.otter.manager.biz.remote.impl.NodeMBeanServiceImpl.getAttribute(NodeMBeanServiceImpl.java:189)
    at com.alibaba.otter.manager.biz.remote.impl.NodeMBeanServiceImpl.getHeapMemoryUsage(NodeMBeanServiceImpl.java:88)
    at com.alibaba.otter.manager.web.home.module.screen.NodeInfo.execute(NodeInfo.java:46)
    at com.alibaba.otter.manager.web.home.module.screen.NodeInfo$$FastClassByCGLIB$$e4987a33.invoke(<generated>)
    at net.sf.cglib.reflect.FastMethod.invoke(FastMethod.java:53)
    at com.alibaba.citrus.service.moduleloader.impl.adapter.MethodInvoker.invoke(MethodInvoker.java:70)
    at com.alibaba.citrus.service.moduleloader.impl.adapter.DataBindingAdapter.executeAndReturn(DataBindingAdapter.java:41)
    at com.alibaba.citrus.turbine.pipeline.valve.PerformScreenValve.performScreenModule(PerformScreenValve.java:111)
    at com.alibaba.citrus.turbine.pipeline.valve.PerformScreenValve.invoke(PerformScreenValve.java:74)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.turbine.pipeline.valve.PerformActionValve.invoke(PerformActionValve.java:73)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210)
    at com.alibaba.citrus.service.pipeline.impl.valve.ChooseValve.invoke(ChooseValve.java:98)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210)
    at com.alibaba.citrus.service.pipeline.impl.valve.LoopValve.invokeBody(LoopValve.java:105)
    at com.alibaba.citrus.service.pipeline.impl.valve.LoopValve.invoke(LoopValve.java:83)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.turbine.pipeline.valve.CheckCsrfTokenValve.invoke(CheckCsrfTokenValve.java:123)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.otter.manager.web.webx.valve.AuthContextValve.invoke(AuthContextValve.java:124)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.turbine.pipeline.valve.AnalyzeURLValve.invoke(AnalyzeURLValve.java:126)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.turbine.pipeline.valve.SetLoggingContextValve.invoke(SetLoggingContextValve.java:66)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.turbine.pipeline.valve.PrepareForTurbineValve.invoke(PrepareForTurbineValve.java:52)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invokeNext(PipelineImpl.java:157)
    at com.alibaba.citrus.service.pipeline.impl.PipelineImpl$PipelineContextImpl.invoke(PipelineImpl.java:210)
    at com.alibaba.citrus.webx.impl.WebxControllerImpl.service(WebxControllerImpl.java:43)
    at com.alibaba.citrus.webx.impl.WebxRootControllerImpl.handleRequest(WebxRootControllerImpl.java:53)
    at com.alibaba.citrus.webx.support.AbstractWebxRootController.service(AbstractWebxRootController.java:165)
    at com.alibaba.citrus.webx.servlet.WebxFrameworkFilter.doFilter(WebxFrameworkFilter.java:152)
    at com.alibaba.citrus.webx.servlet.FilterBean.doFilter(FilterBean.java:147)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at com.alibaba.citrus.webx.servlet.SetLoggingContextFilter.doFilter(SetLoggingContextFilter.java:61)
    at com.alibaba.citrus.webx.servlet.FilterBean.doFilter(FilterBean.java:147)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:559)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:365)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:748)

诊断

用btrace拦截manager的javax.management.remote.JMXConnectorFactory.connect方法，打印入参：

[service:jmx:rmi://10.10.10.1/jndi/rmi://10.10.10.1:2090/mbean, null, ]
[service:jmx:rmi://10.10.10.2/jndi/rmi://10.10.10.2:2090/mbean, null, ]

以上分别为打开正常、异常页面时的入参。

在manager上telnet 报错的node 是通的：

[publish@VM_centos btrace]$ telnet 10.10.10.2 2090
Trying 10.10.10.2...
Connected to 10.10.10.2.
Escape character is '^]'.

问题

所以这说明不是hostname的问题，也不是网络的问题。请问下还有什么可能吗？谢谢

joooohnli commented 6 years ago

更新

补充说明：manager node1是在同台机器上，所以还是不能排除hostname的问题。不过，node2上hostname -i是正常显示ip的，不知道还有什么情况。

另外，查看jmx server注册代码JmxConnectorServerFactoryBean:

    private String              SERVER_URL           = "service:jmx:rmi://127.0.0.1:{0}/jndi/rmi://127.0.0.1:{0}/mbean";

这里为什么不注册为在manager创建node时填写的ip？

HugoDo commented 3 years ago

这个项目是不维护了么？