[Bug] [Module Name] Bug title Bug Flink on Yarn task stops, error message is reported, version 3.2.1

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Apache License 2.0

12.69k stars 4.58k forks source link

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened

dolphinscheduler 停止报错信息。 flink 任务停止不不掉报错

What you expected to happen

dolphinscheduler 停止报错信息，[ERROR] 2024-07-27 19:48:49.939 +0800 - Kill yarn application [[application_1722077591560_0002]] failed org.apache.dolphinscheduler.common.shell.AbstractShell$ExitCodeException: 24/07/27 19:48:48 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://hdp03:8198/ws/v2/timeline/, clusterId=yarn_cluster 24/07/27 19:48:48 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hdp01/10.7.0.100:8050 24/07/27 19:48:49 INFO client.AHSProxy: Connecting to Application History server at hdp02/10.7.0.102:10200 24/07/27 19:48:49 INFO impl.YarnClientImpl: Killed application application_1722077591560_0002

at org.apache.dolphinscheduler.common.shell.AbstractShell.runCommand(AbstractShell.java:205)
at org.apache.dolphinscheduler.common.shell.AbstractShell.run(AbstractShell.java:118)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execute(ShellExecutor.java:125)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:103)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:86)
at org.apache.dolphinscheduler.common.utils.OSUtils.exeShell(OSUtils.java:345)
at org.apache.dolphinscheduler.common.utils.OSUtils.exeCmd(OSUtils.java:334)
at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.execYarnKillCommand(YarnApplicationManager.java:89)
at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.killApplication(YarnApplicationManager.java:48)
at org.apache.dolphinscheduler.plugin.task.api.utils.ProcessUtils.cancelApplication(ProcessUtils.java:192)
at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.doKill(TaskInstanceKillOperationFunction.java:100)
at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.operate(TaskInstanceKillOperationFunction.java:69)
at org.apache.dolphinscheduler.server.worker.rpc.TaskInstanceOperatorImpl.killTask(TaskInstanceOperatorImpl.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.dolphinscheduler.extract.base.server.ServerMethodInvokerImpl.invoke(ServerMethodInvokerImpl.java:41)
at org.apache.dolphinscheduler.extract.base.server.JdkDynamicServerHandler.lambda$processReceived$0(JdkDynamicServerHandler.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

[ERROR] 2024-07-27 19:48:49.940 +0800 - Cancel application failed: 24/07/27 19:48:48 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://hdp03:8198/ws/v2/timeline/, clusterId=yarn_cluster 24/07/27 19:48:48 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hdp01/10.7.0.100:8050 24/07/27 19:48:49 INFO client.AHSProxy: Connecting to Application History server at hdp02/10.7.0.102:10200 24/07/27 19:48:49 INFO impl.YarnClientImpl: Killed application application_1722077591560_0002

[INFO] 2024-07-27 19:48:49.941 +0800 - Begin to kill process process, pid is : 6705 [INFO] 2024-07-27 19:48:49.942 +0800 - Success kill task: 36_36, pid: 6705 [INFO] 2024-07-27 19:48:49.942 +0800 - kill task by cancelApplication, taskInstanceId: 36

How to reproduce

[ERROR] 2024-07-27 19:48:49.939 +0800 - Kill yarn application [[application_1722077591560_0002]] failed org.apache.dolphinscheduler.common.shell.AbstractShell$ExitCodeException: 24/07/27 19:48:48 INFO impl.TimelineReaderClientImpl: Initialized TimelineReader URI=http://hdp03:8198/ws/v2/timeline/, clusterId=yarn_cluster 24/07/27 19:48:48 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hdp01/10.7.0.100:8050 24/07/27 19:48:49 INFO client.AHSProxy: Connecting to Application History server at hdp02/10.7.0.102:10200 24/07/27 19:48:49 INFO impl.YarnClientImpl: Killed application application_1722077591560_0002

at org.apache.dolphinscheduler.common.shell.AbstractShell.runCommand(AbstractShell.java:205)
at org.apache.dolphinscheduler.common.shell.AbstractShell.run(AbstractShell.java:118)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execute(ShellExecutor.java:125)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:103)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:86)
at org.apache.dolphinscheduler.common.utils.OSUtils.exeShell(OSUtils.java:345)
at org.apache.dolphinscheduler.common.utils.OSUtils.exeCmd(OSUtils.java:334)
at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.execYarnKillCommand(YarnApplicationManager.java:89)
at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.killApplication(YarnApplicationManager.java:48)
at org.apache.dolphinscheduler.plugin.task.api.utils.ProcessUtils.cancelApplication(ProcessUtils.java:192)
at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.doKill(TaskInstanceKillOperationFunction.java:100)
at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.operate(TaskInstanceKillOperationFunction.java:69)
at org.apache.dolphinscheduler.server.worker.rpc.TaskInstanceOperatorImpl.killTask(TaskInstanceOperatorImpl.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.dolphinscheduler.extract.base.server.ServerMethodInvokerImpl.invoke(ServerMethodInvokerImpl.java:41)
at org.apache.dolphinscheduler.extract.base.server.JdkDynamicServerHandler.lambda$processReceived$0(JdkDynamicServerHandler.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Anything else

at org.apache.dolphinscheduler.common.shell.AbstractShell.runCommand(AbstractShell.java:205)
at org.apache.dolphinscheduler.common.shell.AbstractShell.run(AbstractShell.java:118)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execute(ShellExecutor.java:125)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:103)
at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:86)
at org.apache.dolphinscheduler.common.utils.OSUtils.exeShell(OSUtils.java:345)
at org.apache.dolphinscheduler.common.utils.OSUtils.exeCmd(OSUtils.java:334)
at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.execYarnKillCommand(YarnApplicationManager.java:89)
at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.killApplication(YarnApplicationManager.java:48)
at org.apache.dolphinscheduler.plugin.task.api.utils.ProcessUtils.cancelApplication(ProcessUtils.java:192)
at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.doKill(TaskInstanceKillOperationFunction.java:100)
at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.operate(TaskInstanceKillOperationFunction.java:69)
at org.apache.dolphinscheduler.server.worker.rpc.TaskInstanceOperatorImpl.killTask(TaskInstanceOperatorImpl.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.dolphinscheduler.extract.base.server.ServerMethodInvokerImpl.invoke(ServerMethodInvokerImpl.java:41)
at org.apache.dolphinscheduler.extract.base.server.JdkDynamicServerHandler.lambda$processReceived$0(JdkDynamicServerHandler.java:108)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Version

3.2.x

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

at org.apache.dolphinscheduler.common.shell.AbstractShell.runCommand(AbstractShell.java:205) at org.apache.dolphinscheduler.common.shell.AbstractShell.run(AbstractShell.java:118) at org.apache.dolphinscheduler.common.shell.ShellExecutor.execute(ShellExecutor.java:125) at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:103) at org.apache.dolphinscheduler.common.shell.ShellExecutor.execCommand(ShellExecutor.java:86) at org.apache.dolphinscheduler.common.utils.OSUtils.exeShell(OSUtils.java:345) at org.apache.dolphinscheduler.common.utils.OSUtils.exeCmd(OSUtils.java:334) at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.execYarnKillCommand(YarnApplicationManager.java:89) at org.apache.dolphinscheduler.plugin.task.api.am.YarnApplicationManager.killApplication(YarnApplicationManager.java:48) at org.apache.dolphinscheduler.plugin.task.api.utils.ProcessUtils.cancelApplication(ProcessUtils.java:192) at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.doKill(TaskInstanceKillOperationFunction.java:100) at org.apache.dolphinscheduler.server.worker.runner.operator.TaskInstanceKillOperationFunction.operate(TaskInstanceKillOperationFunction.java:69) at org.apache.dolphinscheduler.server.worker.rpc.TaskInstanceOperatorImpl.killTask(TaskInstanceOperatorImpl.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.dolphinscheduler.extract.base.server.ServerMethodInvokerImpl.invoke(ServerMethodInvokerImpl.java:41) at org.apache.dolphinscheduler.extract.base.server.JdkDynamicServerHandler.lambda$processReceived$0(JdkDynamicServerHandler.java:108) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

apache / dolphinscheduler