apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.79k stars 1.74k forks source link

[Bug] [connector-doris] doris sink config: `doris.batch.size` not meeting expectations #7277

Closed Larborator closed 1 month ago

Larborator commented 1 month ago

Search before asking

What happened

image

I found when the row reaches the size, the data of cached not write to server immediately. It depends on when checkpoint executed. Is this meet expectations?

SeaTunnel Version

2.3.5

SeaTunnel Config

{
    "env": {
        "parallelism": 1,
        "job.mode": "STREAMING",
        "job.name": "SeaTunnel",
        "checkpoint.interval": 5000,
        "execution.checkpoint.mode": "exactly-once"
    },
    "source": [{
        "plugin_name": "Kafka",
        "startup_mode": "LATEST",
        "commit_on_checkpoint": false,
        ...
    }],
    "sink": [{
        "plugin_name": "Doris",
        "fenodes": "",
        "username": "",
        "password": "",
        "table.identifier": "",
        "sink.enable-2pc": "false",
        "sink.label-prefix": "test",
        "doris.batch.size": 1,
        "doris.config": {
            "format": "json",
            "read_json_by_line": "true"
        }
    }]
}

Running Command

bin/start-seatunnel-flink-13-connector-v2.sh

Error Exception

no error exception

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

Code of Conduct