search

hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
263 stars 149 forks source link

Does this connector support spark 3? #305

Open flxzaj opened 3 years ago

flxzaj commented 3 years ago

Hi, I ran the connector successfully on spark 2.4.5, but failed on spark 3.0.0. Is there any support for spark 3? Thanks

nicolaszhang commented 3 years ago

I met the same issue, any update on this?

dvdgnzlz-maths commented 3 years ago

Hi everyone,

I try to create DeltaTable and use Apache Atlas: I have been able to create from a DeltaTable a table in Hive. CREATE EXTERNAL TABLE delta_tbl(date int, delay int, distance int, origin string, destination string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' WITH SERDEPROPERTIES("path" = "/tmp/departureDelays.delta'") STORED BY 'io.delta.hive.DeltaStorageHandler' LOCATION 'file:///tmp/departureDelays.delta';

I can do it but I'm using Delta Lake 0.6.1. I would like to use the last version (0.7) but I need Spark 3.0. I clone this project to work in this but it have a strong dependency with "spark-atlas-connector-main_2.11". I don't find this project to work with this.

Where can I find the code for "spark-atlas-connector-main_2.11"?

Thanks in advance! Best,

sbbagal13 commented 2 years ago

Anyone found solution for this?