Azure / azure-kusto-spark

Apache Spark Connector for Azure Kusto
Apache License 2.0
77 stars 34 forks source link

Fallback to export mode only if count query fails #261

Closed ag-ramachandran closed 1 year ago

ag-ramachandran commented 1 year ago

The following is the behavior in case readMode is specified

a) If the user does not specify readMode , a count estimate is done and based on the ADX query limits there is switch to ForceDistributedMode (which uses export) or ForceSingleMode (which uses query directly). In the case that this count estimate query fails (query : Table | count), the fallback is to use ForceDistributedMode

b) If the user specifies the readMode as ForceSingleMode the estimate of the query limits is not performed. If the query fails in single mode , it will still fallback to export mode (ForceDistributedMode)

Expected behavior

The fallback to export mode should not happen in case ForceSingleMode is specified and the query fails.