Azure / spark-cdm-connector

MIT License
76 stars 33 forks source link

Error while connecting to model.json using spark pools on azure #113

Closed RithwikChhugani closed 1 year ago

RithwikChhugani commented 1 year ago

I am trying to connect to a model.json file located in a data lake folder. Below is the code that I am using to connect.

val df = spark.read.format("com.microsoft.cdm") .option("storage",".dfs.core.windows.net") .option("cdmModel", "https://....../model.json") .option("entity", "account") .load()

After running the cell with the above code in it, the following is the error.

Screen Shot 2022-11-30 at 3 32 11 pm

Any help would be appreciated.

kecheung commented 1 year ago

You should check the documents. Error message says you are missing the "manifestPath" argument. https://github.com/Azure/spark-cdm-connector/blob/master/documentation/overview.md#common-read-and-write-options

The samples even show the correct usage. https://github.com/Azure/spark-cdm-connector/blob/master/samples/SparkCDMsample.scala