search

cloudera / clusterdock

Apache License 2.0
70 stars 57 forks source link

Error with Hive using oozie #31

Closed Crandel closed 6 years ago

Crandel commented 7 years ago

Hi, I have a problem with hive actions in oozie workflows. I could connect to hive using beeline inside one of the nodes, I could run hive scripts in hue hive query editor, but when I create workflow with this hive script I got this log

2017-05-29 18:21:15,532 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@:start:] Start action [0000002-170529163018023-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-29 18:21:15,536 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@:start:] [***0000002-170529163018023-oozie-oozi-W@:start:***]Action status=DONE
2017-05-29 18:21:15,536 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@:start:] [***0000002-170529163018023-oozie-oozi-W@:start:***]Action updated in DB!
2017-05-29 18:21:15,656 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] Start action [0000002-170529163018023-oozie-oozi-W@hive2-458f] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-05-29 18:21:20,482 INFO org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] checking action, hadoop job ID [job_1496075386293_0005] status [RUNNING]
2017-05-29 18:21:20,484 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] [***0000002-170529163018023-oozie-oozi-W@hive2-458f***]Action status=RUNNING
2017-05-29 18:21:20,484 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] [***0000002-170529163018023-oozie-oozi-W@hive2-458f***]Action updated in DB!
2017-05-29 18:21:29,862 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[node-1.hadoop] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] callback for action [0000002-170529163018023-oozie-oozi-W@hive2-458f]
2017-05-29 18:22:00,240 WARN org.apache.oozie.action.hadoop.Hive2ActionExecutor: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] Exception in check(). Message[java.net.ConnectException: Call From node-1.hadoop/192.168.123.2 to node-1.hadoop:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused]
java.io.IOException: java.net.ConnectException: Call From node-1.hadoop/192.168.123.2 to node-1.hadoop:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:338)
    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:423)
    at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:569)
    at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:604)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:602)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:602)
    at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:612)
    at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:642)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.getRunningJob(JavaActionExecutor.java:1266)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1289)
    at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
    at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Call From node-1.hadoop/192.168.123.2 to node-1.hadoop:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    at org.apache.hadoop.ipc.Client.call(Client.java:1475)
    at org.apache.hadoop.ipc.Client.call(Client.java:1408)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy38.getJobReport(Unknown Source)
    at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
    at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:324)
    ... 20 more
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)
    at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)
    at org.apache.hadoop.ipc.Client.call(Client.java:1447)
    ... 28 more
2017-05-29 18:22:00,241 WARN org.apache.oozie.command.wf.ActionCheckXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] Exception while executing check(). Error Code [  JA006], Message[  JA006: Call From node-1.hadoop/192.168.123.2 to node-1.hadoop:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused]
org.apache.oozie.action.ActionExecutorException:   JA006: Call From node-1.hadoop/192.168.123.2 to node-1.hadoop:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:454)
    at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:434)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1389)
    at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:182)
    at org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:56)
    at org.apache.oozie.command.XCommand.call(XCommand.java:286)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:175)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Call From node-1.hadoop/192.168.123.2 to node-1.hadoop:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor43.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    at org.apache.hadoop.ipc.Client.call(Client.java:1475)
    at org.apache.hadoop.ipc.Client.call(Client.java:1408)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy38.getJobReport(Unknown Source)
    at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
    at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:324)
    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:423)
    at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:569)
    at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:604)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:602)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:602)
    at org.apache.hadoop.mapred.JobClient.getJobInner(JobClient.java:612)
    at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:642)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.getRunningJob(JavaActionExecutor.java:1266)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:1289)
    ... 7 more
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)
    at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)
    at org.apache.hadoop.ipc.Client.call(Client.java:1447)
    ... 28 more
2017-05-29 18:22:00,241 INFO org.apache.oozie.command.wf.ActionCheckXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[0000002-170529163018023-oozie-oozi-W@hive2-458f] Next Retry, Attempt Number [1] in [10,000] milliseconds
2017-05-29 18:22:07,599 INFO org.apache.oozie.command.wf.KillXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[] STARTED WorkflowKillXCommand for jobId=0000002-170529163018023-oozie-oozi-W
2017-05-29 18:22:07,638 INFO org.apache.oozie.command.wf.KillXCommand: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[] ENDED WorkflowKillXCommand for jobId=0000002-170529163018023-oozie-oozi-W
2017-05-29 18:22:07,639 INFO org.apache.oozie.DagEngine: SERVER[node-1.hadoop] USER[oozie] GROUP[-] TOKEN[] APP[My_Workflow] JOB[0000002-170529163018023-oozie-oozi-W] ACTION[] User oozie killed the WF job 0000002-170529163018023-oozie-oozi-W
Back

telnet node-1.hadoop 10020 give me telnet: connect to address 192.168.123.2: Connection refused

dimaspivak commented 6 years ago

Can't repro. Please reopen with details of your system if you're still hitting this.