hortonworks-spark / spark-atlas-connector

A Spark Atlas connector to track data lineage in Apache Atlas
Apache License 2.0
264 stars 149 forks source link

Qualify path with Spark default filesystem when scheme is not specified #263

Closed HeartSaVioR closed 5 years ago

HeartSaVioR commented 5 years ago

Spark uses default filesystem when path is given without scheme. For example, if HDFS is not set up in Spark, /tmp is interpreted as file:///tmp whereas if HDFS is set up in Spark, /tmp is interpreted as hdfs://<hdfs>/tmp.

SAC should follow up the behavior when recording entity to properly create right type of entity.