org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known

[743294668]

RT

Describe the question I used a shell node to execute a sqoop command successfully and finally printed out the command execution return value. It returned 0 but the web showed that the execution failed. What is the problem, how does the DP shell node determine whether the script is successfully executed?

Which version of DolphinScheduler: -[1.2.0]

Additional context Hadoop

[743294668]

[INFO] 2020-04-23 09:37:29.121 - [taskAppId=TASK-48-176-302]:[231] - process id is 742 [INFO] 2020-04-23 09:37:29.122 - [taskAppId=TASK-48-176-302]:[432] - find app id: application_1587605547992_0003 [INFO] 2020-04-23 09:37:29.122 - [taskAppId=TASK-48-176-302]:[236] - yarn log url:application_1587605547992_0003 [ERROR] 2020-04-23 09:37:29.229 org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: Name or service not known java.net.UnknownHostException: null: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getAllByName0(InetAddress.java:1276) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) at org.apache.dolphinscheduler.common.utils.HttpUtils.get(HttpUtils.java:60) at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:377) at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:396) at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.updateState(AbstractCommandExecutor.java:241) at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:170) at org.apache.dolphinscheduler.server.worker.task.shell.ShellTask.handle(ShellTask.java:105) at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.run(TaskScheduleThread.java:142) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [ERROR] 2020-04-23 09:37:29.229 - [taskAppId=TASK-48-176-302]:[410] - yarn applications: [application_1587605547992_0003] status failed : null java.lang.NullPointerException: null at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:380) at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:396) at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.updateState(AbstractCommandExecutor.java:241) at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:170) at org.apache.dolphinscheduler.server.worker.task.shell.ShellTask.handle(ShellTask.java:105) at org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread.run(TaskScheduleThread.java:142) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [INFO] 2020-04-23 09:37:29.229 org.apache.dolphinscheduler.server.worker.runner.TaskScheduleThread:[156] - task instance id : 302,task final status : FAILURE

[743294668]

想问下解决方法是什么？谢谢

[743294668]

Problem solving：Need Yarn configuration in /opt/dolphinscheduler/install/conf/common/hadoop： yarn.application.status.address yarn.resourcemanager.ha.rm.ids

https://cloud.tencent.com/developer/article/1596505

[dukeshiny]

After I set yarn.resourcemanager.ha.rm.ids=cdh201,cdh202,another error happens,even if the sqoop task succeeded!

[gabrywu]

When running the shell script ,worker find strings like application_1587605547992_0003. Then worker think the shell script startup a yarn job , it will monitor the status of the application usig HttpUtils . However missing some config , like yarn.application.status.address yarn.resourcemanager.ha.rm.ids, then errors thrown . DolphinScheduler developers already talked about this ,you can subscribe the dev mail list, and can find what you want to know. I don't agree with this mechanism actually . Maybe you can give better suggestions