microsoft / sql-spark-connector

Apache Spark Connector for SQL Server and Azure SQL
Apache License 2.0
273 stars 116 forks source link

Does bulk upsert data import support? #250

Open melin opened 8 months ago

melin commented 8 months ago

Bulk appended import data. If the primary key exists, update the data. How to import data efficiently? The following operations can be performed in postgresql. Does sqlserver have similar features?

CREATE TABLE if not exists ${tempTableName} (LIKE ${tableName} INCLUDING defaults)

COPY $table FROM STDIN WITH (NULL '\\N', FORMAT CSV, DELIMITER E'${fieldDelimiter}'

insert into tableName (column1.....) select  * from  tempTableName on conflict (id) DO UPDATE SET (......)

https://github.com/melin/datatunnel/blob/master/connectors/jdbc/src/main/scala/com/superior/datatunnel/plugin/jdbc/support/CopyHelper.scala