search

Azure / spark-cdm-connector

MIT License
75 stars 32 forks source link

Error in - Explicit write, creating an entity in a CDM folder based on a pre-defined model #71

Closed sathya1972 closed 3 years ago

sathya1972 commented 3 years ago

UPLOAD CDM FILES FIRST

To run this example, first create a /Models/Contacts folder to your demo container in ADLS gen2,

then upload the provided Contacts.manifest.cdm.json, Person.cdm.json, Entity.cdm.json files

When I was trying to execute following code segment, I get - Entity definition model file "/Contacts/Person.cdm.json" not found., All the directories and files are in place as per above comments.

My directory structure is - adbdemofilesystem / Models / Contacts Files under Contacts are - Contacts.manifest.cdm.json, Person.cdm.json, Entity.cdm.json files

Create the dataframe that matches the CDM definition of the entity, Person

df2 = spark.createDataFrame(spark.sparkContext.parallelize(data2, 1), schema2) (df2.write.format("com.microsoft.cdm") .option("storage", storageAccountName) .option("manifestPath", container + "/Data/Contacts/root.manifest.cdm.json") .option("entity", "Person") .option("entityDefinitionModelRoot", container + "/Models") .option("entityDefinitionPath", "/Contacts/Person.cdm.json/Person") .mode("overwrite") .save())

Py4JJavaError Traceback (most recent call last) /databricks/spark/python/pyspark/sql/utils.py in deco(*a, *kw) 62 try: ---> 63 return f(a, **kw) 64 except py4j.protocol.Py4JJavaError as e:

/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name) 327 "An error occurred while calling {0}{1}{2}.\n". --> 328 format(target_id, ".", name), value) 329 else:

Py4JJavaError: An error occurred while calling o1112.save. : java.lang.IllegalArgumentException: Entity definition model file "/Contacts/Person.cdm.json" not found at com.microsoft.cdm.utils.CDMModelCommon.(CDMModelCommon.scala:85) at com.microsoft.cdm.utils.CDMModelWriter.(CDMModelWriter.scala:38) at com.microsoft.cdm.write.CDMDataSourceWriter.(CDMDataSourceWriter.scala:69) at com.microsoft.cdm.DefaultSource.createWriter(DefaultSource.scala:295) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:277) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380) at py4j.Gateway.invoke(Gateway.java:295) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:251) at java.lang.Thread.run(Thread.java:748)

During handling of the above exception, another exception occurred:

IllegalArgumentException Traceback (most recent call last)

in 34 .option("entityDefinitionModelRoot", container + "/Models") 35 .option("entityDefinitionPath", "/Contacts/Person.cdm.json/Person") ---> 36 .mode("overwrite") 37 .save())
bissont commented 3 years ago

Were you able to run the previous two cells (implicit and explicit write from pre-defined model) from the example?

I just re-ran the sample and it should still work: image

Can you share a similar snapshot?

sathya1972 commented 3 years ago

Yes I was able to successfully run. 

Sent from Yahoo Mail for iPhone

On Monday, March 15, 2021, 7:37 PM, bissont @.***> wrote:

Were you able to run the previous two cells (implicit and explicit write from pre-defined model) from the example?

I just re-ran the sample and it should still work:

Can you share a similar snapshot?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

Here is the snapshot of adls gen 2 folder structure

image

Here is the code snippet

image

bissont commented 3 years ago

That looks similar to what I have.

Can you also take a snapshot of the contents at the container level? Is there a directory called ImplicitTest at the same level Models?

sathya1972 commented 3 years ago

Thank you for the quick response. Yes. I do have directory called ImplicitTest at the same level Models.

image

bissont commented 3 years ago

I'm not sure what the error is then. Can you reach out to asksparkcdm@microsoft.com and we can schedule some time to debug it.

sathya1972 commented 3 years ago

Thank you for your  quick response.

Hello, Aksparkcdm team, Can you please take a look at this issue and help me on this? I am happy to schedule a working session to resolve this. Appreciate your help.

On Tuesday, 16 March, 2021, 11:36:43 am GMT-7, bissont ***@***.***> wrote:

I'm not sure what the error is then. Can you reach out to @.*** and we can schedule some time to debug it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

bissont commented 3 years ago

Hi, What is your e-mail?

From: sathya1972 @.> Sent: Tuesday, March 16, 2021 12:48 PM To: Azure/spark-cdm-connector @.> Cc: Timothy Bisson @.>; Comment @.> Subject: Re: [Azure/spark-cdm-connector] Error in - Explicit write, creating an entity in a CDM folder based on a pre-defined model (#71)

Thank you for your quick response.

Hello, Aksparkcdm team, Can you please take a look at this issue and help me on this? I am happy to schedule a working session to resolve this. Appreciate your help.

On Tuesday, 16 March, 2021, 11:36:43 am GMT-7, bissont @.<mailto:@.>> wrote:

I'm not sure what the error is then. Can you reach out to @.<mailto:@.> and we can schedule some time to debug it.

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fspark-cdm-connector%2Fissues%2F71%23issuecomment-800555704&data=04%7C01%7CTimothy.Bisson%40microsoft.com%7C3414b377c56849c9a10708d8e8b45d1f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637515208647489430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BepZvqWwKgtj%2FFOD928u6iIKVU2kq6HXiI0AD6GohGU%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAD4OPZWQOXQTVFB4GI4RTLTD6YV5ANCNFSM4ZDYAYFA&data=04%7C01%7CTimothy.Bisson%40microsoft.com%7C3414b377c56849c9a10708d8e8b45d1f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637515208647489430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aEAY%2B9eddOyXANfCCeaKi1fZ60ApfXqpHgpkbvbWCoM%3D&reserved=0.

sathya1972 commented 3 years ago

My email ids -1 @. @.

Thanks  Sent from Yahoo Mail for iPhone

On Tuesday, March 16, 2021, 1:10 PM, bissont @.***> wrote:

Hi, What is your e-mail?

From: sathya1972 @.> Sent: Tuesday, March 16, 2021 12:48 PM To: Azure/spark-cdm-connector @.> Cc: Timothy Bisson @.>; Comment @.> Subject: Re: [Azure/spark-cdm-connector] Error in - Explicit write, creating an entity in a CDM folder based on a pre-defined model (#71)

Thank you for your quick response.

Hello, Aksparkcdm team, Can you please take a look at this issue and help me on this? I am happy to schedule a working session to resolve this. Appreciate your help.

On Tuesday, 16 March, 2021, 11:36:43 am GMT-7, bissont @.<mailto:@.>> wrote:

I'm not sure what the error is then. Can you reach out to @.<mailto:@.> and we can schedule some time to debug it.

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

- You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fspark-cdm-connector%2Fissues%2F71%23issuecomment-800555704&data=04%7C01%7CTimothy.Bisson%40microsoft.com%7C3414b377c56849c9a10708d8e8b45d1f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637515208647489430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BepZvqWwKgtj%2FFOD928u6iIKVU2kq6HXiI0AD6GohGU%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAD4OPZWQOXQTVFB4GI4RTLTD6YV5ANCNFSM4ZDYAYFA&data=04%7C01%7CTimothy.Bisson%40microsoft.com%7C3414b377c56849c9a10708d8e8b45d1f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637515208647489430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aEAY%2B9eddOyXANfCCeaKi1fZ60ApfXqpHgpkbvbWCoM%3D&reserved=0.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

sathya1972 commented 3 years ago

I sent an invite for this evening.

Tim

From: samhitha sathya @.> Sent: Tuesday, March 16, 2021 12:47 PM To: Azure/spark-cdm-connector @.>; Azure/spark-cdm-connector @.>; Ask Spark CDM @.> Cc: Author @.>; @. Subject: [EXTERNAL] Re: [Azure/spark-cdm-connector] Error in - Explicit write, creating an entity in a CDM folder based on a pre-defined model (#71)

Thank you for your quick response.

Hello, Aksparkcdm team,

Can you please take a look at this issue and help me on this? I am happy to schedule a working session to resolve this. Appreciate your help.

On Tuesday, 16 March, 2021, 11:36:43 am GMT-7, bissont @.**@.>> wrote:

I'm not sure what the error is then. Can you reach out to @.**@.> and we can schedule some time to debug it.

- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fspark-cdm-connector%2Fissues%2F71%23issuecomment-800510873&data=04%7C01%7CTimothy.Bisson%40microsoft.com%7Cf57a4fd751c54eae21a508d8e8b44fcc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637515208433849319%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=H11Vmtq%2BW2hXo6icUz%2F%2FpfeHZgByvZdfZsqPfiyHW3o%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADXUCDBPC7SXAQMHO7QFCPLTD6QLVANCNFSM4ZDYAYFA&data=04%7C01%7CTimothy.Bisson%40microsoft.com%7Cf57a4fd751c54eae21a508d8e8b44fcc%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637515208433859282%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=W5UJNChWO6j1Ba%2BozqwlZPUdzNlo%2F67IzXDtoRSCQew%3D&reserved=0.

sathya1972 commented 3 years ago

Tim, Thank you for the quick response. I just see below email. I do not see meeting invite.  Am I missing anything? Thanks. On Tuesday, 16 March, 2021, 01:25:32 pm GMT-7, Timothy Bisson @.***> wrote:

I sent an invite for this evening.

 

Tim

 

From: samhitha sathya @.> Sent: Tuesday, March 16, 2021 12:47 PM To: Azure/spark-cdm-connector @.>; Azure/spark-cdm-connector @.>; Ask Spark CDM @.> Cc: Author @.>; @. Subject: [EXTERNAL] Re: [Azure/spark-cdm-connector] Error in - Explicit write, creating an entity in a CDM folder based on a pre-defined model (#71)

 

Thank you for your  quick response.

 

 

Hello, Aksparkcdm team,

 

Can you please take a look at this issue and help me on this? I am happy to schedule a working session to resolve this. Appreciate your help.

 

 

On Tuesday, 16 March, 2021, 11:36:43 am GMT-7, bissont @.***> wrote:

 

 

I'm not sure what the error is then. Can you reach out @.*** and we can schedule some time to debug it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

sathya1972 commented 3 years ago

thank you for your help. I am closing this issue.