The jar provided in releases is originally built and designed for Azure Synapse. The connector is free to use and usage with third party applications will be provided "as-is" with no guarantee of support or it working with your platform. However, the current code is open sourced for contributions should you feel there are improvements that can be made.
If you want to use Databricks then, you will have to build the jar or use the jars we built in the releases. Credential passthrough will not work. As mentioned in #108, "I have received a reply from a Databricks team, they have informed that a credentials pass-through cannot be used in a scheduled tasks, so the problem is in Databricks, not library."
Credential passthrough
As referenced in #134, credential passthrough is a Synapse specific feature. Use app registration or SAS token auth if you are not using Synapse.
CDM Connector save change
If you upgrade from Spark 2 to Spark 3, the CDM connector save behaves differently. If the entity or the manifest does not exist, when doing a dataframe write operation with SaveMode.Append or SaveMode.Overwrite, it will throw something like NoSuchTableException: Manifest doesn't exist. root.manifest.cdm.json.
Solution is to remove the .mode(SaveMode.Append/Overwrite) option.
Jar "doesn't work" / java.lang.NoClassDefFoundError
See the first point. If you have a Spark cluster and then got similar errors, then you probably used the wrong verson. Some of the said classes exclusively exist within Spark 2 or 3, therefore you get an error. Below table shows the mapping of jar version to Spark version. Run this Scala code com.microsoft.cdm.BuildInfo.version if you don't know what version of the CDM connector you have.
Third Party Platforms
The jar provided in releases is originally built and designed for Azure Synapse. The connector is free to use and usage with third party applications will be provided "as-is" with no guarantee of support or it working with your platform. However, the current code is open sourced for contributions should you feel there are improvements that can be made.
All releases will be made here https://github.com/Azure/spark-cdm-connector/releases and not on Maven.
Example
If you want to use Databricks then, you will have to build the jar or use the jars we built in the releases. Credential passthrough will not work. As mentioned in #108, "I have received a reply from a Databricks team, they have informed that a credentials pass-through cannot be used in a scheduled tasks, so the problem is in Databricks, not library."
Credential passthrough
As referenced in #134, credential passthrough is a Synapse specific feature. Use app registration or SAS token auth if you are not using Synapse.
CDM Connector save change
If you upgrade from Spark 2 to Spark 3, the CDM connector save behaves differently. If the entity or the manifest does not exist, when doing a dataframe write operation with
SaveMode.AppendorSaveMode.Overwrite, it will throw something likeNoSuchTableException: Manifest doesn't exist. root.manifest.cdm.json.Solution is to remove the
.mode(SaveMode.Append/Overwrite)option.Jar "doesn't work" / java.lang.NoClassDefFoundError
See the first point. If you have a Spark cluster and then got similar errors, then you probably used the wrong verson. Some of the said classes exclusively exist within Spark 2 or 3, therefore you get an error. Below table shows the mapping of jar version to Spark version. Run this Scala code
com.microsoft.cdm.BuildInfo.versionif you don't know what version of the CDM connector you have.Example scenarios:
Reading a table gives: java.util.NoSuchElementException
See: Spark 3.3: Reading a table gives: java.util.NoSuchElementException: None.get #138.