Closed pp-akursar closed 11 months ago
@microsoft-github-policy-service agree company="PulsePoint"
@pp-akursar The change looks good. can u attach test results on 3.4.0.
Please review and build! :) Highly needed
Test results on Spark 3.4.0 - Pass.
Run :
/*
val test_obj = new SparkConnTest("connectoradmin", "
Results: Passed test_gci_twoPartName_owar : Entered Tablename is mssqlspark.test_gci_twoPartName_owar Operation Overwrite, append and read test_gci_twoPartName_owar : Passed test_gci_tbNameInBracket_owar : Entered Table name is [test_gci_tbNameInBracket_owar] Operation Overwrite, append and read test_gci_tbNameInBracket_owar : Passed test_gci_tabLock_write : Entered test_gci_tabLock_write : Passed test_gci_secureURL_write : Entered test_gci_secureURL_write : Passed test_gci_reordered_columns : Entered test_gci_reordered_columns : Created table test_gci_reordered_columns : Append succcessful test_gci_reordered_columns : Read back table and confirmed data is added succcessful test_gci_reordered_columns : Reordered Write overwrite with truncate test_gci_reordered_columns : Reordered write append test_gci_reordered_columns : Reordered Write overwrite without truncate Passed test_write_parallel : Entered test_write_parallel : Passed test_gci_empty_dataframe : Entered test_gci_empty_dataframe : Passed test_gci_read_write : Entered test_basic_read_write : Passed test_gci_read_write : Passed test_gci_null_values : Entered test_gci_null_values : Passed test_gci_append_rows : Entered test_gci_append_rows : Passed test_gci_truncate_table : Entered test_gci_truncate_table : Passed test_gci_case_sensitivity : Entered test_gci_case_sensitivity : exit test_gci_precision_scale : Entered test_gci_precision_scale : exit test_isolation_level : Entered test_isolation_level : READ_UNCOMMITTED succeded test_isolation_level : READ_COMMITTED succeded test_isolation_level : REPEATABLE_READ succeded test_isolation_level : SNAPShort write start test_isolation_level : SNAPShort write done test_isolation_level : SNAPShort read done test_isolation_level : SNAPShort 5 5 Assert counts test_isolation_level : SNAPSHOT succeded test_isolation_level : isoLevel = ONE test_isolation_level : isoLevel = NONE Exception test_isolation_level : all done test_isolation_level : exit test_gci_limit_escape : Multiple read test Entered test_gci_limit_escape : Passed test_gci_threePartName_owar : Entered Tablename is testconn.mssqlspark.test_gci_threePartName_owar Operation Overwrite, append and read test_gci_threePartName_owar : Passed
Test Pass: Reliability Mode on.
/*
Test :
test_obj.test_sqlmaster_reliable_connector()
Results : Pass test_isolation_level : SNAPSHOT succeded test_isolation_level : isoLevel = NONE test_isolation_level : isoLevel = NONE Exception test_isolation_level : all done test_isolation_level : exit test_gci_limit_escape : Multiple read test Entered test_gci_limit_escape : Passed test_gci_threePartName_owar : Entered Tablename is testconn.mssqlspark.test_gci_threePartName_owar Operation Overwrite, append and read test_gci_threePartName_owar : Passed test_gci_twoPartName_owar : Entered Tablename is mssqlspark.test_gci_twoPartName_owar Operation Overwrite, append and read test_gci_twoPartName_owar : Passed test_gci_tbNameInBracket_owar : Entered Table name is [test_gci_tbNameInBracket_owar] Operation Overwrite, append and read test_gci_tbNameInBracket_owar : Passed test_gci_tabLock_write : Entered test_gci_tabLock_write : Passed test_gci_secureURL_write : Entered test_gci_secureURL_write : Passed test_gci_reordered_columns : Entered test_gci_reordered_columns : Created table test_gci_reordered_columns : Append succcessful test_gci_reordered_columns : Read back table and confirmed data is added succcessful test_gci_reordered_columns : Reordered Write overwrite with truncate test_gci_reordered_columns : Reordered write append test_gci_reordered_columns : Reordered Write overwrite without truncate Passed test_write_parallel : Entered test_write_parallel : Passed test_gci_empty_dataframe : Entered test_gci_empty_dataframe : Passed test_gci_read_write : Entered test_basic_read_write : Passed test_gci_read_write : Passed test_gci_null_values : Entered test_gci_null_values : Passed test_gci_append_rows : Entered test_gci_append_rows : Passed Results : Pass
test_isolation_level : SNAPSHOT succeded test_isolation_level : isoLevel = NONE test_isolation_level : isoLevel = NONE Exception test_isolation_level : all done test_isolation_level : exit test_gci_limit_escape : Multiple read test Entered test_gci_limit_escape : Passed test_gci_threePartName_owar : Entered Tablename is testconn.mssqlspark.test_gci_threePartName_owar Operation Overwrite, append and read test_gci_threePartName_owar : Passed test_gci_twoPartName_owar : Entered Tablename is mssqlspark.test_gci_twoPartName_owar Operation Overwrite, append and read test_gci_twoPartName_owar : Passed test_gci_tbNameInBracket_owar : Entered Table name is [test_gci_tbNameInBracket_owar] Operation Overwrite, append and read test_gci_tbNameInBracket_owar : Passed test_gci_tabLock_write : Entered test_gci_tabLock_write : Passed test_gci_secureURL_write : Entered test_gci_secureURL_write : Passed test_gci_reordered_columns : Entered test_gci_reordered_columns : Created table test_gci_reordered_columns : Append succcessful test_gci_reordered_columns : Read back table and confirmed data is added succcessful test_gci_reordered_columns : Reordered Write overwrite with truncate test_gci_reordered_columns : Reordered write append test_gci_reordered_columns : Reordered Write overwrite without truncate Passed test_write_parallel : Entered test_write_parallel : Passed test_gci_empty_dataframe : Entered test_gci_empty_dataframe : Passed test_gci_read_write : Entered test_basic_read_write : Passed test_gci_read_write : Passed test_gci_null_values : Entered test_gci_null_values : Passed test_gci_append_rows : Entered test_gci_append_rows : Passed
Nice! thanks for reacting this quickly.
@shivsood , can you tell us when the beta will be released and available through maven? We are wondering whether we should revert back to DBR 12.2 for now or if we can just wait a couple more days and switch to the new connector.
And can we assume that for 3.3 it will stay 1.3.0-BETA?
Also, it would be nice if the README could be updated in these two places (even for the previous release):
There are three version sets of the connector available through Maven, a 2.4.x, a 3.0.x and a 3.1.x compatible version.
And 👍 for your work!
Hello, Also wondering when the newest version 1.4.0 for Spark 3.4 will be available on Maven Central - I can't see it on Databricks for Download and it's highly needed!
Thanks :)
@pp-akursar / @shivsood When are we planning to GA release this version 1.4.0
to support Apache Spark 3.4
? And when can we expect support for Apache Spark 3.5.X
? as all previous version has lots of vulnerabilities.
It highly needed, You can also redirect me to concerning person who can better answer.
Changes similar to those of the previous upgrade to 3.3 at https://github.com/microsoft/sql-spark-connector/pull/197
Fixes https://github.com/microsoft/sql-spark-connector/issues/227, specifically
java.lang.NoSuchMethodError: 'org.apache.spark.sql.types.StructType org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(java.sql.ResultSet, org.apache.spark.sql.jdbc.JdbcDialect, boolean)'