search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.88k stars 4.63k forks source link

docker compose deploy dolphinscheduler how to run python script #16386

Closed lgb861213 closed 3 months ago

lgb861213 commented 3 months ago

Search before asking

Description

docker compose deploy dolphinscheduler how to run python script

when I create the python work definition, and test run it, but it failed. the error message is following: [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.021 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] -

[WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.021 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] - Load task instance plugin [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.021 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] - *** [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.022 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[207] - Send task status RUNNING_EXECUTION master: 172.18.0.6:1234 [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.022 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[214] - TenantCode: root check successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[217] - WorkflowInstanceExecDir: /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520 check successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[224] - Create TaskChannel: org.apache.dolphinscheduler.plugin.task.python.PythonTaskChannel successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[229] - Download resources successfully: ResourceContext(resourceItemMap={}) [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[232] - Download upstream files: [] successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[236] - Task plugin instance: PYTHON create successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.p.t.p.PythonTask:[75] - Initialize python task params { "localParams" : [ ], "varPool" : [ ], "rawScript" : "\nprint(\" this is test python code\")", "resourceList" : [ ] } [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[240] - Success initialized task plugin instance successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[243] - Set taskVarPool: null successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] -

[WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] - * Execute task instance ***** [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] - *** [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.023 +0800 o.a.d.p.t.p.PythonTask:[165] - raw python script : print(" this is test python code") [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.024 +0800 o.a.d.p.t.p.PythonTask:[131] - tenantCode :root, task dir:/tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520 [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.024 +0800 o.a.d.p.t.p.PythonTask:[134] - generate python script file:/tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520/py_1519_1520.py [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.024 +0800 o.a.d.p.t.p.PythonTask:[141] - #-- encoding=utf8 --

print(" this is test python code") [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.024 +0800 o.a.d.p.t.a.s.BaseLinuxShellInterceptorBuilder:[64] - Final Shell file is: [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.024 +0800 o.a.d.p.t.a.s.BaseLinuxShellInterceptorBuilder:[65] - ** Script Content * [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.025 +0800 o.a.d.p.t.a.s.BaseLinuxShellInterceptorBuilder:[67] - #!/bin/bash BASEDIR=$(cd dirname $0; pwd) cd $BASEDIR ${PYTHON_LAUNCHER} /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520/py_1519_1520.py [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.025 +0800 o.a.d.p.t.a.s.BaseLinuxShellInterceptorBuilder:[68] - ** Script Content * [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.025 +0800 o.a.d.p.t.a.s.BaseShellInterceptor:[46] - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520/1519_1520.sh [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:06.031 +0800 o.a.d.p.t.a.AbstractCommandExecutor:[154] - process start, process id is: 5066 [WI-0][TI-1520] - [INFO] 2024-07-29 16:09:06.661 +0800 o.a.d.s.w.r.l.TaskInstanceExecutionRunningEventAckListenFunction:[47] - Receive TaskInstanceExecutionRunningEventAck: TaskInstanceExecutionRunningEventAck(taskInstanceId=1520, success=true) [WI-0][TI-1520] - [INFO] 2024-07-29 16:09:06.663 +0800 o.a.d.s.w.r.l.TaskInstanceExecutionInfoEventAckListenFunction:[49] - Receive TaskInstanceExecutionInfoEventAck: TaskInstanceExecutionInfoEventAck(success=true, taskInstanceId=1520) [WI-0][TI-0] - [INFO] 2024-07-29 16:09:07.031 +0800 o.a.d.p.t.a.AbstractTask:[164] - -> /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520/1519_1520.sh: line 4: /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520/py_1519_1520.py: Permission denied [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.032 +0800 o.a.d.p.t.a.AbstractCommandExecutor:[204] - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520, processId:5066 ,exitStatusCode:126 ,processWaitForStatus:true ,processExitValue:126 [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.033 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] -

[WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.033 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] - * Finalize task instance **** [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.033 +0800 o.a.d.p.t.a.l.TaskInstanceLogHeader:[1259] - *** [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.033 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[287] - Upload output files: [] successfully [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.034 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[292] - Send task execute status: FAILURE to master : 172.18.0.6:1234 [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.034 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[109] - Remove the current task execute context from worker cache [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.034 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[299] - The current execute mode isn't develop mode, will clear the task execute file: /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520 [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.035 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[314] - Success clear the task execute file: /tmp/dolphinscheduler/exec/process/root/115497484235776/115512988068096_2/1519/1520 [WI-1519][TI-1520] - [INFO] 2024-07-29 16:09:07.035 +0800 o.a.d.s.w.r.WorkerTaskExecutor:[340] - FINALIZE_SESSION [WI-0][TI-1520] - [INFO] 2024-07-29 16:09:07.661 +0800 o.a.d.s.w.r.l.TaskInstanceExecutionFinishEventAckListenFunction:[48] - Receive TaskInstanceExecutionFinishEventAck: TaskInstanceExecutionFinishEventAck(taskInstanceId=1520, success=true)

how to fix it

Documentation Links

https://dolphinscheduler.apache.org/en-us/docs/3.2.2/guide/task/python

Are you willing to submit a PR?

Code of Conduct

lgb861213 commented 3 months ago

refer https://github.com/apache/dolphinscheduler/discussions/15085 and resolve it