java.lang.NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkData;)V

[whoisvolos]

Good day. Can you help me pls find the reason of issue? Im trying to use connector and Im facing an issue with java.lang.NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkData;)V. I already found some issues, but my configuration as I guess differs.

Im using

spark.read.parquet(location)
      .write
      .mode(SaveMode.Overwrite)
      .format("com.microsoft.sqlserver.jdbc.spark")
      .option("driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver")
      .option("tableLock", true)
      .option("url", connectionString)
      .option("dbtable", tmpTableFullName)
      .option("user", getUsername)
      .option("password", getPassword)
      .save()

Spark is 3.0.3, Scala - 2.12 JDBC - 8.4.1.jre8 Connector version is 1.1.0

Stack trace follows

Caused by: java.lang.NoSuchMethodError: com.microsoft.sqlserver.jdbc.SQLServerBulkCopy.writeToServer(Lcom/microsoft/sqlserver/jdbc/ISQLServerBulkData;)V
        at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.bulkWrite(BulkCopyUtils.scala:110)
        at com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.savePartition(BulkCopyUtils.scala:58)
        at com.microsoft.sqlserver.jdbc.spark.SingleInstanceWriteStrategies$.$anonfun$write$2(BestEffortSingleInstanceStrategy.scala:43)
        at com.microsoft.sqlserver.jdbc.spark.SingleInstanceWriteStrategies$.$anonfun$write$2$adapted(BestEffortSingleInstanceStrategy.scala:42)
        at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2(RDD.scala:994)
        at org.apache.spark.rdd.RDD.$anonfun$foreachPartition$2$adapted(RDD.scala:994)
        at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2154)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:127)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:463)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:466)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[whoisvolos]

UPD: Im doing sbt assembly and I found interesting thing

1. Idea shows that 8.4.1.jre contains this method, I can navigate there by given stack trace
2. But when I vim-ed assembly jar and found SQLServerBulkCopy class I was unable to find method "writeToServer" with given signature (ISQLServerBulkData), only with ResultSet.

[luxu1-ms]

Can you check your jdbc version? I guess the spark mssql connector you use is compatible with jdbc 8.4.1, but the jdbc in your environment is jdbc-7. Need to remove old jar from environment and replace it by mssql-jdbc-8.4.1..jar

[luxu1-ms]

closing as old/answered. Please open if still an issue after using the right JDBC version.

[whoisvolos]

Yes, seems so, thank you!