search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
264 stars 149 forks source link

invalid relationshipDef: avro_schema_associatedEntities: end type 1: DataSet, end type 2: spark_table #70

Closed krisss85 closed 5 years ago

krisss85 commented 6 years ago

we receive this message when creating a table in spark. This is with Atlas 1.0 and Spark 2.3.1

18/09/20 14:32:53 WARN RestAtlasClient: Failed to create entities
org.apache.atlas.AtlasServiceException: Metadata service API org.apache.atlas.AtlasClientV2$API_V2@4539545f failed with status 400 (Bad Request) Response Body ({"errorCode":"ATLAS-400-00-036","errorMessage":"invalid relationshipDef: avro_schema_associatedEntities: end type 1: DataSet, end type 2: spark_table"})
    at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:395)
    at org.apache.atlas.AtlasBaseClient.callAPIWithResource(AtlasBaseClient.java:323)
    at org.apache.atlas.AtlasBaseClient.callAPI(AtlasBaseClient.java:211)
    at org.apache.atlas.AtlasClientV2.createEntities(AtlasClientV2.java:305)
    at com.hortonworks.spark.atlas.RestAtlasClient.doCreateEntities(RestAtlasClient.scala:68)
    at com.hortonworks.spark.atlas.AtlasClient$class.createEntities(AtlasClient.scala:42)
    at com.hortonworks.spark.atlas.RestAtlasClient.createEntities(RestAtlasClient.scala:31)
    at com.hortonworks.spark.atlas.sql.SparkCatalogEventProcessor.process(SparkCatalogEventProcessor.scala:64)
    at com.hortonworks.spark.atlas.sql.SparkCatalogEventProcessor.process(SparkCatalogEventProcessor.scala:30)
    at com.hortonworks.spark.atlas.AbstractEventProcessor$$anonfun$eventProcess$1.apply(AbstractEventProcessor.scala:67)
    at com.hortonworks.spark.atlas.AbstractEventProcessor$$anonfun$eventProcess$1.apply(AbstractEventProcessor.scala:66)
    at scala.Option.foreach(Option.scala:257)
    at com.hortonworks.spark.atlas.AbstractEventProcessor.eventProcess(AbstractEventProcessor.scala:66)
    at com.hortonworks.spark.atlas.AbstractEventProcessor$$anon$1.run(AbstractEventProcessor.scala:39)

We don't know how to solve this. Any idea?

merlintang commented 6 years ago

@krisss85 can you give me more details to reproduce this ?

krisss85 commented 6 years ago

hi @merlintang, Thanks for looking into that. Spark and Atlas installed from Hortonworks HDP 3 distribution. spark-atlas-connector from the v0.1.0-2.3-1.0.0 release. To reproduce it is fairly simple. Just try to save any DataFrame as a table in that configuration.

merlintang commented 6 years ago

@krisss85 thanks for your report, let me try to reproduce this and give you feedback asap.

bolkedebruin commented 6 years ago

@merlintang any update on this? we are hitting the same issue

krisss85 commented 6 years ago

I've just tested the PR in Hortonworks HDP3 environment with Atlas 1.0. It solved the issue.