Nice to have: a "Worker" node configured with PENTAHO PDI ( CARTE ).

Arturo-Penas-Rial commented 1 year ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

I am testing version 3.1 of DS, and the changes introduced are impressive. I would like to make a POC with DS to introduce it in my organization, but I need your help. I have tried to create an image from apache/dolphinscheduler-worker, but with no success.

I need help to have a "worker" node that has PENTAHO CARTE installed (in order to invoke the ETL process execution via HTTP). An example of an image with CARTE is "nentangso/pentaho-kettle-carte:latest"

Excuse me, but I need your help. I have tried to search for manuals, blogs, without success. I think DS is an excellent product (it should be more popular than Apache Airflow!), but I need CARTE integration. Can you help me, provide a model image, or provide documentation to achieve it?

Use case

Users who use ETL from PENTAHO PDI, who want to use DS, need a worker node with PENTAHO PDI with its services exposed through CARTE (HTTP).

Especially useful to compete against Apache Airflow which has integrations with NiFi and PENTAHO. Would allow users with traditional ETL to use DS orchestration

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

github-actions[bot] commented 1 year ago

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

kezhenxu94 commented 1 year ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

I am testing version 3.1 of DS, and the changes introduced are impressive. I would like to make a POC with DS to introduce it in my organization, but I need your help. I have tried to create an image from apache/dolphinscheduler-worker, but with no success.

I need help to have a "worker" node that has PENTAHO CARTE installed (in order to invoke the ETL process execution via HTTP). An example of an image with CARTE is "nentangso/pentaho-kettle-carte:latest"

Just write your own Dockerfile and base from the apache/dolphinscheduler-worker, build and push to you registry and use it. This is a rather common sense to extend a Docker image just like how we build apache/dolphinscheduler-worker from openjdk base image.

Excuse me, but I need your help. I have tried to search for manuals, blogs, without success. I think DS is an excellent product (it should be more popular than Apache Airflow!), but I need CARTE integration. Can you help me, provide a model image, or provide documentation to achieve it?

zhongjiajie commented 1 year ago

and we have related docs about it, you can see https://dolphinscheduler.apache.org/en-us/docs/3.0.1/user_doc/contribute/development-environment-setup.html section "Docker image build", we have "create custom Docker images" about how to create custom images base on official release images

Especially useful to compete against Apache Airflow which has integrations with NiFi and PENTAHO

And BTW, I check the airflow document, it seem they do not support both NIFI and PENTAHO

zhongjiajie commented 1 year ago

Hi @Arturo-Penas-Rial are you still there?

Arturo-Penas-Rial commented 1 year ago

Hi @zhongjiajie

Yes :-) still tying. Thanks for your response and availability. Now the news ...

I reach to modify and activate a worker image including PENTAHO PDI (docker pull arturopenas/dolphinscheduler-pdi-worker:latest),
but I cannot access from outside the container. Can I reuse the default port 1235, to expose internal HTTP services?
I would like to check if the image customized is Ok, so ... is there any mechanism to check if the image is OK for DS?
The state of a task definition, named "Need fault tolerance", that’s mean? I have a deployment based on docker compose with only one worker node.

I fell DS is a great product!, but it will be easier to adopt if a set of worker images with integrations like PENTAHO PDI, NIFI, would be available.

Thanks again.

Arturo-Penas-Rial commented 1 year ago

One more :-) @zhongjiajie ...

When I try to execute a DAG the following error log occurs. I believe there is a miss configuration, can you guide me?

[LOG-PATH]: /opt/dolphinscheduler/logs/20221018/7222558634432_2-6-9.log, [HOST]: Host{address='172.20.0.5:1234', ip='172.20.0.5', port=1234} [INFO] 2022-10-18 14:12:40.511 +0000 - Begin to pulling task [INFO] 2022-10-18 14:12:40.525 +0000 - Begin to initialize task [INFO] 2022-10-18 14:12:40.525 +0000 - Set task startTime: Tue Oct 18 14:12:40 UTC 2022 [INFO] 2022-10-18 14:12:40.525 +0000 - Set task envFile: /opt/dolphinscheduler/conf/dolphinscheduler_env.sh [INFO] 2022-10-18 14:12:40.526 +0000 - Set task appId: 6_9 [INFO] 2022-10-18 14:12:40.526 +0000 - End initialize task [INFO] 2022-10-18 14:12:40.527 +0000 - Set task status to TaskExecutionStatus{code=1, desc='running'} [ERROR] 2022-10-18 14:12:40.601 +0000 - Task execute failed, due to meet an exception org.apache.dolphinscheduler.plugin.task.api.TaskException: TenantCode: Inquilino #2 doesn't exist at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionCheckerUtils.checkTenantExist(TaskExecutionCheckerUtils.java:62) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.beforeExecute(WorkerTaskExecuteRunnable.java:186) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:149) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [INFO] 2022-10-18 14:12:40.603 +0000 - Get a exception when execute the task, will send the task execute result to master, the current task execute result is TaskExecutionStatus{code=6, desc='failure'}

Note: When I access "workflow definition" and I try to execute the DAG the following error message appear "master does not exist", but all process are up and running.

Arturo-Penas-Rial commented 1 year ago

The name of a workflow definition can change (modified)?

zhongjiajie commented 1 year ago

The name of a workflow definition can change (modified)?

It supports, you can change the workflow name when you saving the workflow definition, for more detail see gif

how-to-rename2

zhongjiajie commented 1 year ago

I reach to modify and activate a worker image including PENTAHO PDI

can you share what is your Dockerfile for this PENTAHO PDI worker docker images?

Can I reuse the default port 1235

You can not, 1235 is for workers to communicate with other servers, it is being used and will throw an error because is be used.

so ... is there any mechanism to check if the image is OK for DS

you can check in tab "monitor" to see whether the worker is up, and then run a simple task to find our it work or not

check-worker

zhongjiajie commented 1 year ago

And I wonder whether PENTAHO PDI has a client or CLI interface that we can submit our job and execute them? Actually, we only need to submit the job in DolphinScheduler worker, it can run this job in PENTAHO PDI itself running an application with a web UI interface in DolphinScheduler worker maybe not be the best practice

caishunfeng commented 1 year ago

The state of a task definition, named "Need fault tolerance", that’s mean? I have a deployment based on docker compose with only one worker node.

The Need fault tolerance is the state means that these task instances will rerun again by tolerance (the old one was finished and create a new one for same workflow instance). This fault tolerance mechanism is triggered when the worker server down (task instances tolerance) or master server down(workflow instances tolerance).

see https://dolphinscheduler.apache.org/en-us/docs/latest/user_doc/architecture/design.html, the Fault-Tolerant Design part.

caishunfeng commented 1 year ago

One more :-) @zhongjiajie ...

When I try to execute a DAG the following error log occurs. I believe there is a miss configuration, can you guide me?

[LOG-PATH]: /opt/dolphinscheduler/logs/20221018/7222558634432_2-6-9.log, [HOST]: Host{address='172.20.0.5:1234', ip='172.20.0.5', port=1234} [INFO] 2022-10-18 14:12:40.511 +0000 - Begin to pulling task [INFO] 2022-10-18 14:12:40.525 +0000 - Begin to initialize task [INFO] 2022-10-18 14:12:40.525 +0000 - Set task startTime: Tue Oct 18 14:12:40 UTC 2022 [INFO] 2022-10-18 14:12:40.525 +0000 - Set task envFile: /opt/dolphinscheduler/conf/dolphinscheduler_env.sh [INFO] 2022-10-18 14:12:40.526 +0000 - Set task appId: 6_9 [INFO] 2022-10-18 14:12:40.526 +0000 - End initialize task [INFO] 2022-10-18 14:12:40.527 +0000 - Set task status to TaskExecutionStatus{code=1, desc='running'} [ERROR] 2022-10-18 14:12:40.601 +0000 - Task execute failed, due to meet an exception org.apache.dolphinscheduler.plugin.task.api.TaskException: TenantCode: Inquilino #2 doesn't exist at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionCheckerUtils.checkTenantExist(TaskExecutionCheckerUtils.java:62) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.beforeExecute(WorkerTaskExecuteRunnable.java:186) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:149) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) [INFO] 2022-10-18 14:12:40.603 +0000 - Get a exception when execute the task, will send the task execute result to master, the current task execute result is TaskExecutionStatus{code=6, desc='failure'}

Note: When I access "workflow definition" and I try to execute the DAG the following error message appear "master does not exist", but all process are up and running.

Hi @Arturo-Penas-Rial I try at local and it can create a new tenant by default config. I think you can try to execute cmd sudo useradd -g user-group new-user in the worker node. Generally, it is recommended to use the created tenant in ds after the system has created the tenant.

nielifeng commented 1 year ago

Can you provide me with your e-mail or other contact information?

Arturo-Penas-Rial commented 1 year ago

Many thanks for your quick response!, I will follow your guidelines. My email is arturo.penas@gmail.com.

William-GuoWei commented 1 year ago

Hi, I think here are some solutions for you:

DS worker is not like Airflow, it can support install Kettle Client on Worker, but not a docker of Kettle with DS client. So you need to have a kettle docker besides the worker server (DS Worker can run on the same server on DS Master).
You can easily use Http Task in DS to trigger Kettle job with http request.
I am thinking about add a Kettle task in DS, and that will be easier for you to use kettle as a docker to run ETL jobs and close it after ETL finish.

For now, If your ETL job is on cloud, I think maybe you need create 3 task for Kettle,

Create a shell or python task of start a docker with Kettle server
Create a Http task or shell task to trigger Kettle job with http request
Create a shell or python task of close the docker with Kettle server

I hope that can help you.

Arturo-Penas-Rial commented 1 year ago

Hi @William-GuoWei ... Let me check if I understood :-)

In Airflow, the worker node can contain both PENTAHO PDI and CARTE, I managed to do it and it works. However, I would like to use DS, but I still can't get a worker node with PENTAHO PDI and CARTE.

What I am trying to do is to extend the DS worker image, to incorporate PENTAHO. Is it possible? I do not want to have a docker of Kettle with DS client inside a "worker".

From your previous message I understand that the above would not be possible, so I should create a docker node with PENTAHO PDI and CARTE (not integrating the DS orchestration) and access it through the worker? If so, how do I schedule a task (ETL) since the PENTAHO node would be out of scope?

My ETLs are in local server, and the idea was to migrate it to a DS "worker" node, sharing a directory with the ETL files and from the DS agenda invoke via HTTP the execution in CARTE of those ETLs.

I haven't found any communication to guide me on how to customize the "worker". It would be great if DS could have images with pre-installed typocos products like for example PENTAHO!

How do you think I can move forward?

Thanks and have a nice day!

Arturo-Penas-Rial commented 1 year ago

Well I reach to extend the worker node image, also PENTAHO Carte is running on this node, but DS find as unhealty state ...:-( The worker node log does not show errors! Can you give me some guidance?

apache / dolphinscheduler