search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
264 stars 149 forks source link

invalid insance creaion/updaion parameers passed: spark.column.table mandatory attribute value missing in type spark_column #306

Open Deepakparashar14 opened 4 years ago

Deepakparashar14 commented 4 years ago

hi , i am using this connector in hdp3.1 and spark 2.3 and sac has been configured from the ambari by enabling this feature,so couple of things its not able to do is:

  1. not showing the columns of the table in atlas spark_db entities and also the spark_column.
  2. no lineage of the database and tables . 3.no reference of tables in hive db created using spark process for that database created from spark. IMG_20200812_212821 IMG_20200813_134613
Magister89 commented 3 years ago

Similar problem on HDP 3.1.5, when I try to execute a spark-sql CTAS or INSERT INTO TABLE AS SELECT from an Hive table. Atlas and Hive Hook are working. Lineage pops up when loading a CSV and save into an Hive table, as expected.

killerdongbo commented 3 years ago

spark table level lineage problem i had solved,but the spark column level lineage problem shows like this: org.apache.atlas.exception.AtlasBaseException: Invalid instance creation/updation parameters passed : spark_column.table: mandatory attribute value missing in type spark_column.