apache / inlong

Apache InLong - a one-stop, full-scenario integration framework for massive data
https://inlong.apache.org/
Apache License 2.0
1.4k stars 530 forks source link

[Feature][Manager] Add Apache Doris load node management #5043

Closed dockerzhang closed 2 years ago

dockerzhang commented 2 years ago

Description

Add Apache Doris Node Management, you can refer to: https://inlong.apache.org/docs/next/design_and_concept/how_to_extend_data_node_for_manager

Dashboard: https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_dashboard

Use case

No response

Are you willing to submit PR?

Code of Conduct

Loveca commented 2 years ago

Motivation

To surport the ability to Doris data integration, we need to add Apache Doris Load Node for management

Design

The design mainly follows the document Manager Plugin

  1. Add corresponding SinkType enumeration in enumeration type org.apache.inlong.manager.common.Enums.
  2. In org.apache.inlong.manager.common.Pojo.Sink,create folder path,create the corresponding entity class.
  3. In the org.Apache.Inlong.Manager.Service.Sink path, created under the corresponding tools
  4. Support data source to LoadNode conversion function, reference code org.Apache. Inlong.Manager.Service.Sort.Util.LoadNodeUtils

Implementation

  1. Add DORIS in enumeration type org.apache.inlong.manager.common.enums.SinkType

  2. Create folder "Doris" in org.apache.inlong.manager.common.pojo.sink, and create corresponding entity class:

    • DorisSink
    • DorisSinkDTO
    • DorisSinkRequest
    • DorisColumnInfo
    • DorisTableInfo
  3. Create folder "doris" in org.apache.inlong.manager.service.sink and implement the class:

    • DorisSinkOperator
  4. Add createLoadNode function in org.apache.inlong.manager.service.sort.util.LoadNodeUtils, it is like as follows:

    public static DorisLoadNode createLoadNode(DorisSink dorisSink, List<FieldInfo> fieldInfos,
            List<FieldRelation> fieldRelations, Map<String, String> properties){
                 \\TODO 
      }
Loveca commented 2 years ago

@dockerzhang hi, can you assign this to me?

dockerzhang commented 2 years ago

@Loveca LGTM

healchow commented 2 years ago

Duplicate with #6232, close this one.