apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.74k stars 4.59k forks source link

[Improvement][task-java/task-spark] Task-Java and Task-Spark add Convert Parameter Placeholders for args #16250

Closed lanxing2 closed 1 month ago

lanxing2 commented 3 months ago

Search before asking

Description

Currently, for task-java
ParameterUtils.convertParameterPlaceholders will replace placeholder for the Java Code if Java Task run in the Java Mode.
However if run the Java Task in Jar Mode, I cannot pass Parameters to the task, because in the mainArgs and jvmArgs do not have ParameterUtils.convertParameterPlaceholders to replace the placeholders with parameter value.
If we enable placeholder replacement for args, JavaTask in Jar Mode can get the value of Parameters in args.
Similiarily, for task-spark
ParameterUtils.convertParameterPlaceholders will replace placeholder for the SQL Code if Java Task run in the SQLMode.
However if run the Spark Task in other Mode, cannot pass the parameters to the task, because mainArgs and otherArgs do not have ParameterUtils.convertParameterPlaceholders to replace the placeholders with parameter value.
If we enable placeholder replacement for args, SparkTask in Mode other than SQL can get the value of Parameters in args.
I purpose this Improvement because in my team, we save some common data such in project parameters.
And use these parameters in different processes and tasks, then I notice JavaTask in Jar Mode and Spark Task not in SQL Mode cannot use these parameters. I actually add placeholder replacement for Java Task in Jar Mode in our teams' cluster.
Hope the core member agree with this purposal, I can submit PR for this feature/improvement.

Are you willing to submit a PR?

Code of Conduct

lanxing2 commented 3 months ago

In JavaTask, MainArgs and JVMArgs are use directly without Placeholder Replacement https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-java/src/main/java/org/apache/dolphinscheduler/plugin/task/java/JavaTask.java#L169-L170 https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-java/src/main/java/org/apache/dolphinscheduler/plugin/task/java/JavaTask.java#L190-L191

lanxing2 commented 3 months ago

In SparkTask, MainArgs and others are use directly without Placeholder Replacement https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java#L198 https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java#L180

lanxing2 commented 3 months ago

In SparkTask, MainArgs and others are use directly without Placeholder Replacement https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java#L198 https://github.com/apache/dolphinscheduler/blob/dev/dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java#L180

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 month ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.