apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.04k stars 1.82k forks source link

[Feature][Core] Support config validate command #7739

Open Hisoka-X opened 1 month ago

Hisoka-X commented 1 month ago

Search before asking

Description

Support validate config file by https://github.com/apache/seatunnel/blob/71b8b943903ec8ff3868f104e5dc979fccf34bd5/seatunnel-core/seatunnel-starter/src/main/java/org/apache/seatunnel/core/starter/seatunnel/command/SeaTunnelConfValidateCommand.java#L42

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

Cheun99 commented 1 month ago

pls assign to me !

Cheun99 commented 1 month ago

Hi @Hisoka-X I have some questions?

1. The validate method

Should i not only support command-line submit of job config verification, but also support restapi verification?

2. The degree of validate

2.1 Only checks whether this config file can generate DAG? 2.2 The Required OptionRule configuration of the connector also need to be verified?

Hisoka-X commented 1 month ago

Should i not only support command-line submit of job config verification, but also support restapi verification?

We can support shell script at first, then support restapi with another pr.

2.1 Only checks whether this config file can generate DAG?

Although the feature is called verification, its behavior is closer to pre-execution. We need to execute the code directly until LogicalDag is generated. During the pre-execution process, we can print out a lot of useful information, such as the number and schema of the tables produced by source/transform, and the display of the final generated LogicalDag in log.

2.2 The Required OptionRule configuration of the connector also need to be verified?

after 2.1 finished, the optionrule also be verified.

github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.