scouter-project / scouter

Scouter is an open source APM (Application Performance Management) tool.
Other
2.09k stars 563 forks source link

summary api 호출시 EOFException 발생 #677

Open yuna-seo opened 4 years ago

yuna-seo commented 4 years ago

안녕하세요 현재 시스템에서 모니터링 툴로 scouter를 사용중입니다

다름이 아니고 summary api 호출시에 EOFException이 떨어지는 경우가 종종 발생하여서 질문드립니다

특정 경우에만 발생하는 것을 보니, 예외적인 케이스가 있는것 같은데 혹시 summary api의 end of file 처리는 어떤식으로 동작하는것인지 궁금합니다

그리고 혹시 동일한 케이스가 발생헀을 때, 서비스는 정상적으로 이용할 수 있도록 처리할 수 있는 다른 방법이 있을까요?

e9l10 22:00:14.694 [qtp1755425911-216] ERROR s.w.f.e.m.GenericExceptionMapper[95] - [ErrorStateException] 500 - internal server error - java.io.EOFException, [uri]v1/summary/error/ofType/tomcat_30prd at scouterx.webapp.framework.exception.ErrorState.newException(ErrorState.java:67)

scouterx.webapp.framework.exception.ErrorStateException: java.io.EOFException at scouterx.webapp.framework.exception.ErrorState.newException(ErrorState.java:67) at scouterx.webapp.model.summary.Summary.of(Summary.java:58) at scouterx.webapp.layer.consumer.SummaryConsumer.retrieveErrorSummary(SummaryConsumer.java:148) at scouterx.webapp.layer.service.SummaryService.retrieveErrorSummary(SummaryService.java:59) at scouterx.webapp.layer.controller.SummaryController.retrieveErrorSummary(SummaryController.java:275) at scouterx.webapp.layer.controller.SummaryController.retrieveErrorSummaryByType(SummaryController.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:243) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268) at org.glassfish.jersey.internal.Errors.process(Errors.java:316) at org.glassfish.jersey.internal.Errors.process(Errors.java:298) at org.glassfish.jersey.internal.Errors.process(Errors.java:268) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650) at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:206) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at scouterx.webapp.framework.filter.ReleaseResourceFilter.doFilter(ReleaseResourceFilter.java:44) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at scouterx.webapp.framework.filter.NoCacheFilter.doFilter(NoCacheFilter.java:40) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at scouterx.webapp.framework.filter.CorsFilter.doFilter(CorsFilter.java:65) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at scouterx.webapp.framework.filter.LoggingInitServletFilter.doFilter(LoggingInitServletFilter.java:50) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:521) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:564) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.io.EOFException at scouterx.webapp.framework.client.net.TcpProxy.process(TcpProxy.java:206) at scouterx.webapp.framework.client.net.TcpProxy.process(TcpProxy.java:131) at scouterx.webapp.framework.client.model.TextModel.load(TextModel.java:165) at scouterx.webapp.framework.client.model.TextModel.getText(TextModel.java:119) at scouterx.webapp.framework.client.model.TextModel.getTextIfNullDefault(TextModel.java:99) at scouterx.webapp.model.summary.ErrorSummaryItem.toSummary(ErrorSummaryItem.java:103) at scouterx.webapp.model.summary.Summary.of(Summary.java:56) ... 65 common frames omitted Caused by: java.io.EOFException: null at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at scouter.io.DataInputX.read(DataInputX.java:78) at scouter.io.DataInputX.readBlob(DataInputX.java:107) at scouter.io.DataInputX.readText(DataInputX.java:143) at scouter.lang.value.TextValue.read(TextValue.java:73) at scouter.io.DataInputX.readValue(DataInputX.java:256) at scouter.lang.pack.MapPack.read(MapPack.java:178) at scouter.io.DataInputX.readPack(DataInputX.java:268) at scouterx.webapp.framework.client.net.TcpProxy.lambda$process$0(TcpProxy.java:132) at scouterx.webapp.framework.client.net.TcpProxy.process(TcpProxy.java:197) ... 71 common frames omitted

gunlee01 commented 4 years ago

이 에러는 여러 케이스가 있겠지만 scouter collector와 webapp 간 통신에서 불특정 원인으로 통신이 끊어지는 경우에도 발생할 듯 합니다. (그외 다양한 원인이 있겠지요. 특정 조건의 버그일 수도 있습니다.) 재시도시에도 동일한 계속해서 에러가 발생하는지요? 그렇지 않다면 호출하는 쪽에서 에러 발생시 retry를 해주는 수밖에 없을 듯 합니다.