search

ow2-proactive / scheduling

Multi-platform Scheduling and Workflows Engine
http://www.activeeon.com/workflows-scheduling
GNU Affero General Public License v3.0
62 stars 54 forks source link

Timer already cancelled exception at login #2256

Closed fviale closed 8 years ago

fviale commented 9 years ago

When login into a 6.1 scheduler (from any interface, here the exception is printed from the CLI), the following error message appears :

Error Message: java.lang.IllegalStateException: Timer already cancelled.
Stack Track: java.lang.Throwable: java.lang.IllegalStateException: Timer already cancelled.
        at org.ow2.proactive_grid_cloud_portal.common.Session.getScheduler(Session.java:113)
        at org.ow2.proactive_grid_cloud_portal.scheduler.SchedulerStateRest.checkAccess(SchedulerStateRest.java:1400)
        at org.ow2.proactive_grid_cloud_portal.scheduler.SchedulerStateRest.revisionAndjobsinfo(SchedulerStateRest.java:366)
        at sun.reflect.GeneratedMethodAccessor819.invoke(null)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
        at java.lang.reflect.Method.invoke(null)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
        at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:370)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
        at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(null)
Caused by: java.lang.Throwable: Timer already cancelled.
        at java.util.Timer.sched(null)
        at java.util.Timer.schedule(null)
        at org.ow2.proactive.scheduler.core.SchedulerFrontendState.renewUserSession(SchedulerFrontendState.java:244)
        at org.ow2.proactive.scheduler.core.SchedulerFrontendState.connect(SchedulerFrontendState.java:211)
        at org.ow2.proactive.scheduler.core.SchedulerFrontend.connect(SchedulerFrontend.java:296)
        at sun.reflect.GeneratedMethodAccessor777.invoke(null)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
        at java.lang.reflect.Method.invoke(null)
        at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
        at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
        at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
        at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
        at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
        at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
        at org.objectweb.proactive.Service.serve(Service.java:125)
        at org.ow2.proactive.scheduler.core.SchedulerFrontend.runActivity(SchedulerFrontend.java:868)
        at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
        at java.lang.Thread.run(null)
        at org.ow2.proactive.scheduler.core.SchedulerFrontend.connect(pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.core.SchedulerFrontend_699205c7-14e43bd4fa3--7fff--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000)
        at (...)(null)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
        at pa.stub.org.ow2.proactive.scheduler.core._StubSchedulerFrontend.connect(_StubSchedulerFrontend.java)
        at org.ow2.proactive.scheduler.authentication.SchedulerAuthentication.login(SchedulerAuthentication.java:119)
        at sun.reflect.GeneratedMethodAccessor776.invoke(null)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
        at java.lang.reflect.Method.invoke(null)
        at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
        at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
        at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
        at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
        at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
        at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
        at org.objectweb.proactive.Service.serve(Service.java:125)
        at org.ow2.proactive.authentication.AuthenticationImpl.runActivity(AuthenticationImpl.java:245)
        at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
        at java.lang.Thread.run(null)
        at org.ow2.proactive.scheduler.common.SchedulerAuthenticationInterface.login([pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.authentication.SchedulerAuthentication_699205c7-14e43bd4fa3--7ffe--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000, pnp://frrdscp01.rd.loreal:64738/SCHEDULER])
        at (...)(null)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
        at pa.stub.org.ow2.proactive.scheduler.common._StubSchedulerAuthenticationInterface.login(_StubSchedulerAuthenticationInterface.java)
        at org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface.init(SchedulerProxyUserInterface.java:145)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(null)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
        at java.lang.reflect.Method.invoke(null)
        at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
        at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
        at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
        at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
        at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
        at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
        at org.objectweb.proactive.Service.blockingServeOldest(Service.java:206)
        at org.objectweb.proactive.Service.blockingServeOldest(Service.java:181)
        at org.objectweb.proactive.Service.fifoServing(Service.java:146)
        at org.objectweb.proactive.core.body.ActiveBody$FIFORunActive.runActivity(ActiveBody.java:273)
        at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
        at java.lang.Thread.run(null)
        at org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface.init(pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface_699205c7-14e4899f501-690c--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000)
        at (...)(null)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
        at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
        at pa.stub.org.ow2.proactive.scheduler.common.util._StubSchedulerProxyUserInterface.init(_StubSchedulerProxyUserInterface.java)
        at org.ow2.proactive_grid_cloud_portal.common.SchedulerRMProxyFactory.connectToScheduler(SchedulerRMProxyFactory.java:73)
        at org.ow2.proactive_grid_cloud_portal.common.Session.connectToScheduler(Session.java:92)
        at org.ow2.proactive_grid_cloud_portal.common.Session.getScheduler(Session.java:109)
        ... 40 more

inspecting the Scheduler log, the following exception appears all of the sudden, and then continuously appears in response to login attemps from the SCOM agent :

No other exceptions appear when users try to connect via the tradition web interface.

[2015-07-01 05:51:03,994 INFO     o.o.p.s.c.SchedulingService] SCOM successfully connected !
[2015-07-01 05:51:04,006 WARN          o.e.j.s.ServletHandler] 
javax.servlet.ServletException: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException
    at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Unknown Source)
Caused by: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException
    at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:157)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    ... 34 more
Caused by: java.lang.NullPointerException
    at org.jboss.resteasy.core.MediaTypeMap$CachedMediaTypeAndClass.<init>(MediaTypeMap.java:273)
    at org.jboss.resteasy.core.MediaTypeMap$CachedMediaTypeAndClass.<init>(MediaTypeMap.java:261)
    at org.jboss.resteasy.core.MediaTypeMap.getPossible(MediaTypeMap.java:404)
    at org.jboss.resteasy.spi.ResteasyProviderFactory.resolveMessageBodyWriter(ResteasyProviderFactory.java:2107)
    at org.jboss.resteasy.spi.ResteasyProviderFactory.getMessageBodyWriter(ResteasyProviderFactory.java:2096)
    at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:62)
    at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:153)
    ... 45 more
[2015-07-01 05:51:04,008 WARN          o.e.j.s.ServletHandler] /workflow-catalog/wc/login
org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException
    at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:157)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
    at org.jboss.resteasy.core.MediaTypeMap$CachedMediaTypeAndClass.<init>(MediaTypeMap.java:273)
    at org.jboss.resteasy.core.MediaTypeMap$CachedMediaTypeAndClass.<init>(MediaTypeMap.java:261)
    at org.jboss.resteasy.core.MediaTypeMap.getPossible(MediaTypeMap.java:404)
    at org.jboss.resteasy.spi.ResteasyProviderFactory.resolveMessageBodyWriter(ResteasyProviderFactory.java:2107)
    at org.jboss.resteasy.spi.ResteasyProviderFactory.getMessageBodyWriter(ResteasyProviderFactory.java:2096)
    at org.jboss.resteasy.core.ServerResponseWriter.writeNomapResponse(ServerResponseWriter.java:62)
    at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:153)
    ... 45 more
lpellegr commented 9 years ago

What is SCOM agent?

fviale commented 9 years ago

SCOM is a monitoring framework made by Microsoft and similar to Nagios. I wrote a SCOM agent which logs into the scheduler and submits a job to verify that everything works properly.

fviale commented 9 years ago

Restarting the scheduler solves the problem, but it will reoccur most likely, maybe it's due to shiro, and sessions timeout, maybe the connections were not logged out properly ?

The issue never appeared before in this scheduler, only after the SCOM agent was created and start to submit jobs periodically.

youribonnaffe commented 9 years ago

Could be related to #2005 maybe. Is it reproducible or did it happen once? How often does this SCOM agent logs in?

fviale commented 9 years ago

Ah there is one exception like this, here is the log exceirpt :

[2015-07-01 03:52:08,269 INFO  .p.s.a.SchedulerAuthentication] SCOM is trying to connect
[2015-07-01 03:52:08,270 INFO  .s.a.SchedulerLDAP2LoginModule] Using Login file for fall back authentication at: /opt/proactive/ProActiveWorkflowsScheduling-linux-x64-6.1.0/config/authentication/login.cfg
[2015-07-01 03:52:08,270 INFO  .s.a.SchedulerLDAP2LoginModule] Using Group file for fall back group membership at: /opt/proactive/ProActiveWorkflowsScheduling-linux-x64-6.1.0/config/authentication/group.cfg
[2015-07-01 03:52:08,270 INFO  .p.s.a.SchedulerAuthentication] User SCOM logged successfully
[2015-07-01 03:52:08,270 INFO  .p.s.a.SchedulerAuthentication] user : SCOM
[2015-07-01 03:52:08,272 INFO     o.o.p.s.c.SchedulingService] End of session for user SCOM, id=ActiveObject_org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface_699205c7-14e43bd4fa3--27bf--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000
[2015-07-01 03:52:08,272 INFO     o.o.p.s.c.SchedulingService] SCOM successfully connected !
[2015-07-01 03:52:08,273 INFO     o.o.p.s.c.SchedulingService] User 'SCOM' has disconnect the scheduler !
[2015-07-01 03:52:08,297 WARN  .o.p.w.SchedulerAuthentication] Could not login
java.util.ConcurrentModificationException: java.lang.Throwable: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.ow2.proactive_grid_cloud_portal.scheduler.client.SchedulerRestClient.rebuildServerSideException(SchedulerRestClient.java:542)
    at org.ow2.proactive_grid_cloud_portal.scheduler.client.SchedulerRestClient.access$000(SchedulerRestClient.java:90)
    at org.ow2.proactive_grid_cloud_portal.scheduler.client.SchedulerRestClient$RestClientExceptionHandler.invoke(SchedulerRestClient.java:497)
    at com.sun.proxy.$Proxy91.login(Unknown Source)
    at org.ow2.proactive.workflowcatalog.utils.scheduling.SchedulerProxy.connectToScheduler(SchedulerProxy.java:138)
    at org.ow2.proactive.workflowcatalog.utils.scheduling.SchedulerProxy.<init>(SchedulerProxy.java:62)
    at org.ow2.proactive.workflowcatalog.api.SchedulerProxyFactory.create(SchedulerProxyFactory.java:50)
    at org.ow2.proactive.workflowcatalog.api.WorkflowCatalogSchedulerAuthentication.loginToSchedulerRestApi(WorkflowCatalogSchedulerAuthentication.java:60)
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.commonLogin(SchedulerAuthentication.java:60)
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.login(SchedulerAuthentication.java:41)
    at sun.reflect.GeneratedMethodAccessor831.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.Throwable: null
    at java.util.HashMap$HashIterator.nextEntry(null)
    at java.util.HashMap$KeyIterator.next(null)
    at org.ow2.proactive.scheduler.common.SchedulerUsers.update(SchedulerUsers.java:105)
    at org.ow2.proactive.scheduler.core.SchedulerFrontendState.connect(SchedulerFrontendState.java:213)
    at org.ow2.proactive.scheduler.core.SchedulerFrontend.connect(SchedulerFrontend.java:296)
    at sun.reflect.GeneratedMethodAccessor777.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
    at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
    at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
    at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
    at org.objectweb.proactive.Service.serve(Service.java:125)
    at org.ow2.proactive.scheduler.core.SchedulerFrontend.runActivity(SchedulerFrontend.java:868)
    at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
    at java.lang.Thread.run(null)
    at org.ow2.proactive.scheduler.core.SchedulerFrontend.connect(pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.core.SchedulerFrontend_699205c7-14e43bd4fa3--7fff--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000)
    at (...)(null)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
    at pa.stub.org.ow2.proactive.scheduler.core._StubSchedulerFrontend.connect(_StubSchedulerFrontend.java)
    at org.ow2.proactive.scheduler.authentication.SchedulerAuthentication.login(SchedulerAuthentication.java:119)
    at sun.reflect.GeneratedMethodAccessor776.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
    at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
    at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
    at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
    at org.objectweb.proactive.Service.serve(Service.java:125)
    at org.ow2.proactive.authentication.AuthenticationImpl.runActivity(AuthenticationImpl.java:245)
    at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
    at java.lang.Thread.run(null)
    at org.ow2.proactive.scheduler.common.SchedulerAuthenticationInterface.login([pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.authentication.SchedulerAuthentication_699205c7-14e43bd4fa3--7ffe--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000, pnp://frrdscp01.rd.loreal:64738/SCHEDULER])
    at (...)(null)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
    at pa.stub.org.ow2.proactive.scheduler.common._StubSchedulerAuthenticationInterface.login(_StubSchedulerAuthenticationInterface.java)
    at org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface.init(SchedulerProxyUserInterface.java:183)
    at sun.reflect.GeneratedMethodAccessor774.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
    at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
    at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
    at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
    at org.objectweb.proactive.Service.blockingServeOldest(Service.java:206)
    at org.objectweb.proactive.Service.blockingServeOldest(Service.java:181)
    at org.objectweb.proactive.Service.fifoServing(Service.java:146)
    at org.objectweb.proactive.core.body.ActiveBody$FIFORunActive.runActivity(ActiveBody.java:273)
    at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
    at java.lang.Thread.run(null)
    at org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface.init(pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface_699205c7-14e4715c948-707c--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000)
    at (...)(null)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
    at pa.stub.org.ow2.proactive.scheduler.common.util._StubSchedulerProxyUserInterface.init(_StubSchedulerProxyUserInterface.java)
    at org.ow2.proactive_grid_cloud_portal.common.SchedulerRMProxyFactory.connectToScheduler(SchedulerRMProxyFactory.java:81)
    at org.ow2.proactive_grid_cloud_portal.common.Session.connectToScheduler(Session.java:98)
    at org.ow2.proactive_grid_cloud_portal.scheduler.SchedulerStateRest.login(SchedulerStateRest.java:2206)
    at sun.reflect.GeneratedMethodAccessor779.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(null)
[2015-07-01 03:52:08,305 WARN  .o.p.w.SchedulerAuthentication] Could not login
java.util.ConcurrentModificationException: java.lang.Throwable: null
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.ow2.proactive_grid_cloud_portal.scheduler.client.SchedulerRestClient.rebuildServerSideException(SchedulerRestClient.java:542)
    at org.ow2.proactive_grid_cloud_portal.scheduler.client.SchedulerRestClient.access$000(SchedulerRestClient.java:90)
    at org.ow2.proactive_grid_cloud_portal.scheduler.client.SchedulerRestClient$RestClientExceptionHandler.invoke(SchedulerRestClient.java:497)
    at com.sun.proxy.$Proxy91.login(Unknown Source)
    at org.ow2.proactive.workflowcatalog.utils.scheduling.SchedulerProxy.connectToScheduler(SchedulerProxy.java:138)
    at org.ow2.proactive.workflowcatalog.utils.scheduling.SchedulerProxy.<init>(SchedulerProxy.java:62)
    at org.ow2.proactive.workflowcatalog.api.SchedulerProxyFactory.create(SchedulerProxyFactory.java:50)
    at org.ow2.proactive.workflowcatalog.api.WorkflowCatalogSchedulerAuthentication.loginToSchedulerRestApi(WorkflowCatalogSchedulerAuthentication.java:60)
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.commonLogin(SchedulerAuthentication.java:60)
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.login(SchedulerAuthentication.java:41)
    at sun.reflect.GeneratedMethodAccessor831.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.Throwable: null
    at java.util.HashMap$HashIterator.nextEntry(null)
    at java.util.HashMap$KeyIterator.next(null)
    at org.ow2.proactive.scheduler.common.SchedulerUsers.update(SchedulerUsers.java:105)
    at org.ow2.proactive.scheduler.core.SchedulerFrontendState.connect(SchedulerFrontendState.java:213)
    at org.ow2.proactive.scheduler.core.SchedulerFrontend.connect(SchedulerFrontend.java:296)
    at sun.reflect.GeneratedMethodAccessor777.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
    at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
    at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
    at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
    at org.objectweb.proactive.Service.serve(Service.java:125)
    at org.ow2.proactive.scheduler.core.SchedulerFrontend.runActivity(SchedulerFrontend.java:868)
    at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
    at java.lang.Thread.run(null)
    at org.ow2.proactive.scheduler.core.SchedulerFrontend.connect(pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.core.SchedulerFrontend_699205c7-14e43bd4fa3--7fff--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000)
    at (...)(null)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
    at pa.stub.org.ow2.proactive.scheduler.core._StubSchedulerFrontend.connect(_StubSchedulerFrontend.java)
    at org.ow2.proactive.scheduler.authentication.SchedulerAuthentication.login(SchedulerAuthentication.java:119)
    at sun.reflect.GeneratedMethodAccessor776.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
    at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
    at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
    at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
    at org.objectweb.proactive.Service.serve(Service.java:125)
    at org.ow2.proactive.authentication.AuthenticationImpl.runActivity(AuthenticationImpl.java:245)
    at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
    at java.lang.Thread.run(null)
    at org.ow2.proactive.scheduler.common.SchedulerAuthenticationInterface.login([pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.authentication.SchedulerAuthentication_699205c7-14e43bd4fa3--7ffe--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000, pnp://frrdscp01.rd.loreal:64738/SCHEDULER])
    at (...)(null)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
    at pa.stub.org.ow2.proactive.scheduler.common._StubSchedulerAuthenticationInterface.login(_StubSchedulerAuthenticationInterface.java)
    at org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface.init(SchedulerProxyUserInterface.java:183)
    at sun.reflect.GeneratedMethodAccessor774.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.objectweb.proactive.core.mop.MethodCall.execute(MethodCall.java:355)
    at org.objectweb.proactive.core.body.request.RequestImpl.serveInternal(RequestImpl.java:216)
    at org.objectweb.proactive.core.body.request.RequestImpl.serve(RequestImpl.java:162)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serveInternal(BodyImpl.java:553)
    at org.objectweb.proactive.core.body.BodyImpl$ActiveLocalBodyStrategy.serve(BodyImpl.java:486)
    at org.objectweb.proactive.core.body.AbstractBody.serve(AbstractBody.java:428)
    at org.objectweb.proactive.Service.blockingServeOldest(Service.java:206)
    at org.objectweb.proactive.Service.blockingServeOldest(Service.java:181)
    at org.objectweb.proactive.Service.fifoServing(Service.java:146)
    at org.objectweb.proactive.core.body.ActiveBody$FIFORunActive.runActivity(ActiveBody.java:273)
    at org.objectweb.proactive.core.body.ActiveBody.run(ActiveBody.java:166)
    at java.lang.Thread.run(null)
    at org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface.init(pnp://frrdscp01.rd.loreal:64738/ActiveObject_org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface_699205c7-14e4715c948-707c--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000)
    at (...)(null)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reifyAsSynchronous(AbstractBodyProxy.java:347)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.invokeOnBody(AbstractBodyProxy.java:174)
    at org.objectweb.proactive.core.body.proxy.AbstractBodyProxy.reify(AbstractBodyProxy.java:125)
    at pa.stub.org.ow2.proactive.scheduler.common.util._StubSchedulerProxyUserInterface.init(_StubSchedulerProxyUserInterface.java)
    at org.ow2.proactive_grid_cloud_portal.common.SchedulerRMProxyFactory.connectToScheduler(SchedulerRMProxyFactory.java:81)
    at org.ow2.proactive_grid_cloud_portal.common.Session.connectToScheduler(Session.java:98)
    at org.ow2.proactive_grid_cloud_portal.scheduler.SchedulerStateRest.login(SchedulerStateRest.java:2206)
    at sun.reflect.GeneratedMethodAccessor779.invoke(null)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(null)
    at java.lang.reflect.Method.invoke(null)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(null)
[2015-07-01 03:52:08,306 WARN        o.j.r.c.ExceptionHandler] failed to execute
javax.ws.rs.WebApplicationException: HTTP 500 Internal Server Error
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.logAndThrowHttpException(SchedulerAuthentication.java:76)
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.commonLogin(SchedulerAuthentication.java:70)
    at org.ow2.proactive.workflowcatalog.SchedulerAuthentication.login(SchedulerAuthentication.java:41)
    at sun.reflect.GeneratedMethodAccessor831.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:280)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:234)
    at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:221)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
    at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Unknown Source)
[2015-07-01 03:52:08,323 INFO     o.o.p.s.c.SchedulingService] End of session for user SCOM, id=ActiveObject_org.ow2.proactive.scheduler.common.util.SchedulerProxyUserInterface_699205c7-14e43bd4fa3--27bd--d2bb531c02b153b9-699205c7-14e43bd4fa3--8000
[2015-07-01 03:52:08,324 INFO     o.o.p.s.c.SchedulingService] User 'SCOM' has disconnect the scheduler !
[2015-07-01 03:52:08,391 INFO            o.o.p.s.u.TaskLogger] task 22780000 finished without errors
[2015-07-01 03:52:08,392 INFO            o.o.p.s.u.TaskLogger] task 22780000 result added to job 2278
[2015-07-01 03:52:08,393 INFO             o.o.p.s.u.JobLogger] job 2278 terminated
[2015-07-01 03:52:08,406 INFO             o.o.p.s.u.JobLogger] job 2278 finished tasks 1, total tasks 1, finished true
[2015-07-01 03:52:08,409 INFO            o.o.p.s.u.TaskLogger] task 22840000 scheduling
[2015-07-01 03:52:08,411 INFO             o.o.p.s.u.JobLogger] job 2278 cleaning loggers

The scheduler seems to be able to run jobs afterwards and respond to login sessions from SCOM agent, the complete blocking of the system occurs later on.

Concerning the SCOM agent, It starts a few sessions (around 10) every 15 minutes. This is due to a wrong configuration of my management pack in SCOM as it should log once every 15 minutes. I setup the SCOM monitoring yesterday evening and the problem appeared this morning. Now I've restarted the scheduler, I suppose the problem may reappear in the afternoon. I need to check the Powershell script I'm using to make sure each session is properly logged out.

fviale commented 8 years ago

seems fixed with 6.3.0