I am inserting data into a Azure SQL DB table using bulk inserts with mode "append".
When the table does not exist before the table is created as expected.
When the table does already exist I am getting the following error:
An error occurred while calling o6686.save.
: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema$default$4()Z
at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)
at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)
at com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)
....
I am inserting data into a Azure SQL DB table using bulk inserts with mode "append".
When the table does not exist before the table is created as expected. When the table does already exist I am getting the following error:
An error occurred while calling o6686.save. : java.lang.NoSuchMethodError: org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema$default$4()Z at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305) at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266) at com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79) ....
some searching around, I found the commit https://github.com/apache/spark/commit/702180614900bdaf245a194da0043b8b51de3b4b#diff-c3859e97335ead4b131263565c987d877bea0af3adbd6c5bf2d3716768d2e083 which changed the interface for the getSchema function. But the commit relates only to the master branch is tagged for release 4.0.0.