apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.06k stars 1.83k forks source link

[connectors][hive sink] cannot overwrite to sink hive table #5025

Open bdxmq opened 1 year ago

bdxmq commented 1 year ago

Search before asking

Description

I want to synchronize data to the hive table and need to be able to overwrite the original table data during synchronization. I found that this configuration item is not available

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

ocean-zhc commented 1 year ago

+1

shuiliuxing commented 1 year ago

It is abnormal that hive-sink haven't overwrite mode, hope that future versions can add this feature

xtz123 commented 1 year ago

But the situation I encountered is that sink to hive is covered, and the configuration is as follows: kafka ->hive:

env { execution.parallelism = 1 job.mode = "STREAMING" execution.checkpoint.interval = 10000 execution.checkpoint.data-uri = "obs://ddmc-obs-test/user/flink/checkpoint" }

source {

Kafka { result_table_name = "kafka_name" schema = { fields { name = "string" age = "int" } } format = json topic = "topic3" bootstrap.servers = "x.x.x.x:9092" kafka.config = { client.id = client_2 auto.offset.reset = "latest" enable.auto.commit = "false" } }

}

sink { Hive { table_name = "mydemo.hive_mysql" metastore_uri = "thrift://x.x.x.x:9083" } }

I don't want to overwrite, how should I configure it?

lap50 commented 1 year ago

+1

chenJz1012 commented 1 year ago

+1

Doooing commented 8 months ago

+1

kevin0317qin commented 7 months ago

anyone solve this problem?

frankiewen commented 3 months ago

+1 Did anyone tell me if the new version solves this problem?