microsoft / sql-spark-connector

Apache Spark Connector for SQL Server and Azure SQL
Apache License 2.0
273 stars 116 forks source link

False infos displayed: #257

Open throuch opened 3 months ago

throuch commented 3 months ago

Hello,

when schemaCheckEnabled is set to false we get confusing messages like this one (not exhaustive):

24/05/13 09:51:01 INFO BulkCopyUtils: Spark Dataframe and SQL Server table have differing column nullable configurations at column index 14 DF col XXX nullable config is true Table col XXX nullable config is true

in the class BulkCopyUtils, we have this implementation:

private def assertIfCheckEnabled(
            cond: Boolean, checkEnabled : Boolean,  msg: String): Unit = {
        if(checkEnabled) {
            assertCondition(cond, msg)
        }
        else{
           logInfo(msg)
        }
    }

It's wrong as it displays misleading messages (saying there are differences) when the condition is true. So I suggest this patch:

private def assertIfCheckEnabled(
            cond: Boolean, checkEnabled : Boolean,  msg: String): Unit = {
        if(checkEnabled) {
            assertCondition(cond, msg)
        }
        else if (!cond) {
           logWarn(msg)
        }
    }