search

microsoft / sql-spark-connector

Apache Spark Connector for SQL Server and Azure SQL
Apache License 2.0
273 stars 116 forks source link

append with bulk insert does results in NoSuchMethodError with DB runtime 12.2 LTS while overwrite works #263

Open eschnepel opened 1 month ago

eschnepel commented 1 month ago

I am inserting data into a Azure SQL DB table using bulk inserts with mode "append".

When the table does not exist before the table is created as expected. When the table does already exist I am getting the following error:

An error occurred while calling o6686.save. : java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema$default$4()Z at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305) at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266) at com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79) ....

some searching around, I found the commit https://github.com/apache/spark/commit/702180614900bdaf245a194da0043b8b51de3b4b#diff-c3859e97335ead4b131263565c987d877bea0af3adbd6c5bf2d3716768d2e083 which changed the interface for the getSchema function. But the commit relates only to the master branch is tagged for release 4.0.0.