search

Azure / azure-kusto-spark

Apache Spark Connector for Azure Kusto
Apache License 2.0
77 stars 35 forks source link

Combination of tableCreateOptions as true & adjustSchema results in error Source dataframe has more columns than in ADX #260

Closed ag-ramachandran closed 1 year ago

ag-ramachandran commented 2 years ago

Describe the bug While using the library & using the API dataframe.writeStream, lot of options can be provided while pushing data to ADX. Using the 2 options below results in an error

.option("tableCreateOptions","CreateIfNotExist") 
.option("adjustSchema","GenerateDynamicCsvMapping")

Expectation: When the table doesn't exist, the table should get created. But if it exists, then column name to column name mapping should happen and data should get loaded properly in all the columns in ADX

Observation: When both the options are there and if the table doesn't exist, table is not getting created and also getting error instead of creating the table: "Source dataframe has more columns than in ADX".

The check should be disabled in case the table does not exist!