search

azkaban / azkaban

Azkaban workflow manager.
https://azkaban.github.io
Apache License 2.0
4.46k stars 1.59k forks source link

Azkaban Multiple executor mode error #773

Open gunesh4u opened 7 years ago

gunesh4u commented 7 years ago

Hi,

Recieved following error when configured azkaban to run in multiple executor mode.

azkaban.executor.ExecutorManagerException: azkaban.executor.ExecutorManagerException: java.io.IOException: Error loading flow with exec 19 at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1745) at azkaban.executor.ExecutorManager.submitExecutableFlow(ExecutorManager.java:1041) at azkaban.webapp.servlet.ExecutorServlet.ajaxExecuteFlow(ExecutorServlet.java:885) at azkaban.webapp.servlet.ExecutorServlet.ajaxAttemptExecuteFlow(ExecutorServlet.java:844) at azkaban.webapp.servlet.ExecutorServlet.handleAJAXAction(ExecutorServlet.java:160) at azkaban.webapp.servlet.ExecutorServlet.handleGet(ExecutorServlet.java:84) at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:121) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: azkaban.executor.ExecutorManagerException: java.io.IOException: Error loading flow with exec 19 at azkaban.executor.ExecutorManager.callExecutorServer(ExecutorManager.java:1071) at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1739) ... 21 more Caused by: java.io.IOException: Error loading flow with exec 19 at azkaban.executor.ExecutorManager.callExecutorForJsonObject(ExecutorManager.java:1143) at azkaban.executor.ExecutorManager.callExecutorServer(ExecutorManager.java:1124) at azkaban.executor.ExecutorManager.callExecutorServer(ExecutorManager.java:1068) ... 22 more

I am using current build of 3.0.0 build for Azkaban Web Server and Azkaban Executer server All DB set up has been done for Web server as well as for executers servers as well.

Please suggest?

Thanks, Gunesh P.

diaowenyang commented 6 years ago

@gunesh4u I met the error too.How to fix it ?

gunesh4u commented 6 years ago

Can you share error message/log?

Thanks, Gp

On Fri, 17 Nov 2017 at 1:05 PM, Leno Bill notifications@github.com wrote:

@gunesh4u https://github.com/gunesh4u I met the error too.How to fix it ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/azkaban/azkaban/issues/773#issuecomment-345167688, or mute the thread https://github.com/notifications/unsubscribe-auth/AV2-Wyy7RWuthUGTNB94FqZ4ZLEyttxjks5s3TcpgaJpZM4KaNUY .

-- Sent from Gmail Mobile

diaowenyang commented 6 years ago

@gunesh4u Thanks ！ a

[wherehows@mesos1 bin]$ 2017-11-17 16:08:59 INFO LoginAbstractAzkabanServlet:169 - 10.129.3.1 azkaban "GET /executor projectId=3&project=t3&ajax=executeFlow&flow=test_d3_azkaban&disabled=%5B%5D&failureEmailsOverride=false&successEmailsOverride=false&failureAction=finishCurrent&failureEmails=&successEmails=&notifyFailureFirst=false&notifyFailureLast=false&concurrentOption=skip HTTP/1.1" browser 2017-11-17 16:08:59 INFO ExecutorManager:939 - Submitting execution flow test_d3_azkaban by azkaban 2017-11-17 16:08:59 INFO ExecutionFlowDao:72 - Flow given test_d3_azkaban given id 9 2017-11-17 16:08:59 INFO ExecutorServlet:115 - User has called action execute on 9 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - azkaban.executor.ExecutorManagerException: Error loading flow with exec 9 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.execapp.FlowRunnerManager.submitFlow(FlowRunnerManager.java:324) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.execapp.ExecutorServlet.handleAjaxExecute(ExecutorServlet.java:276) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.execapp.ExecutorServlet.doGet(ExecutorServlet.java:124) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.Server.handle(Server.java:326) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2017-11-17 16:08:59 ERROR ExecutorServlet:279 - Error loading flow with exec 9 azkaban.executor.ExecutorManagerException: Error loading flow with exec 9 at azkaban.execapp.FlowRunnerManager.submitFlow(FlowRunnerManager.java:324) at azkaban.execapp.ExecutorServlet.handleAjaxExecute(ExecutorServlet.java:276) at azkaban.execapp.ExecutorServlet.doGet(ExecutorServlet.java:124) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2017-11-17 16:08:59 ERROR ExecutorManager:1378 - Rolling back executor assignment for execution id:9 azkaban.executor.ExecutorManagerException: java.io.IOException: Error loading flow with exec 9 at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:78) at azkaban.executor.ExecutorApiGateway.callWithExecutable(ExecutorApiGateway.java:43) at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1375) at azkaban.executor.ExecutorManager.submitExecutableFlow(ExecutorManager.java:1013) at azkaban.webapp.servlet.ExecutorServlet.ajaxExecuteFlow(ExecutorServlet.java:943) at azkaban.webapp.servlet.ExecutorServlet.ajaxAttemptExecuteFlow(ExecutorServlet.java:903) at azkaban.webapp.servlet.ExecutorServlet.handleAJAXAction(ExecutorServlet.java:175) at azkaban.webapp.servlet.ExecutorServlet.handleGet(ExecutorServlet.java:95) at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:123) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.io.IOException: Error loading flow with exec 9 at azkaban.executor.ExecutorApiGateway.callForJsonObjectMap(ExecutorApiGateway.java:106) at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:76) ... 24 more 2017-11-17 16:08:59 INFO FlowRunnerManager:818 - # of executing flows: 0 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - azkaban.executor.ExecutorManagerException: azkaban.executor.ExecutorManagerException: java.io.IOException: Error loading flow with exec 9 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1381) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorManager.submitExecutableFlow(ExecutorManager.java:1013) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.webapp.servlet.ExecutorServlet.ajaxExecuteFlow(ExecutorServlet.java:943) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.webapp.servlet.ExecutorServlet.ajaxAttemptExecuteFlow(ExecutorServlet.java:903) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.webapp.servlet.ExecutorServlet.handleAJAXAction(ExecutorServlet.java:175) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.webapp.servlet.ExecutorServlet.handleGet(ExecutorServlet.java:95) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:123) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at javax.servlet.http.HttpServlet.service(HttpServlet.java:770) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.Server.handle(Server.java:326) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - Caused by: azkaban.executor.ExecutorManagerException: java.io.IOException: Error loading flow with exec 9 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:78) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorApiGateway.callWithExecutable(ExecutorApiGateway.java:43) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorManager.dispatch(ExecutorManager.java:1375) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - ... 22 more 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - Caused by: java.io.IOException: Error loading flow with exec 9 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorApiGateway.callForJsonObjectMap(ExecutorApiGateway.java:106) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorApiGateway.callWithExecutionId(ExecutorApiGateway.java:76) 2017-11-17 16:08:59 ERROR StdOutErrRedirect:55 - ... 24 more