search

Azure / spark-cdm-connector

MIT License
76 stars 33 forks source link

[FeatureRequest] Support for Spark 3.4+ and Microsoft Fabric #152

Open gbrueckl opened 9 months ago

gbrueckl commented 9 months ago

Summary of issue

are there any plans to support Spark 3.4 (and later) and more in particular, are there any plans to support Microsoft Fabric

Error stack trace

No response

Platform name

Microsoft Fabric

Spark version

3.4

CDM jar version

NA

What is the format of the data you are trying to read/write?

.parquet

synesthesia commented 1 month ago

As Synapse is now alerting that version Spark 3.3 is deprecated is there a plan to upgrade this component to be 3.4 compatible?

hlenschow commented 1 month ago

@synesthesia we struggle with the same issue, but we have concluded that the cdm-connector is abandoned, and to solve this we have set up Fabric link to OneLake for our Dataverse and Dynamics resources. So we have replaced the cdm read operations, with OneLake read operations.

synesthesia commented 1 month ago

we struggle with the same issue, but we have concluded that the cdm-connector is abandoned, and to solve this we have set up Fabric link to OneLake for our Dataverse and Dynamics resources. So we have replaced the cdm read operations, with OneLake read operations.

@hlenschow sadly we are on a hand-crafted lake built before Fabric was released, and no plans to port

gbrueckl commented 1 month ago

It turned out that Fabric Spark Runtime 1.1 also has the CDM connector installed by default and can be used as a workaround for now. If you migrate to Fabric anyway thats probably the best thing you can do.

This does not help though if you are not on Fabric :/

AndriXS commented 6 days ago

Looking at Fabric runtime 1.2 and 1.3, it doesn't appear to support CDM connector either. Looking forward for spark 3.4 support