CompareSchemaDataFrames and FullOuterJoin

FINRAOS / MegaSparkDiff

A Spark-based data comparison tool at scale which facilitates software development engineers to compare a plethora of pair combinations of possible data sources. Multiple execution modes in multiple environments enable the user to generate a diff report as a Java/Scala-friendly DataFrame or as a file for future use. Comes with out of the box SparkFactory and SparkCompare tools.

https://finraos.github.io/MegaSparkDiff/

Apache License 2.0

48 stars 26 forks source link

CompareSchemaDataFrames and FullOuterJoin #28

Closed aosama closed 6 years ago

aosama commented 6 years ago

Modified SparkCompare.CompareSchemaDataFrames to not need or use tempViewName which makes it more convenient while using DataBricks Notebooks

Added a FullOUterJoin method in spark compare to make it easier to visualize differences while using IDE and console

BlackDuckCoPilot commented 6 years ago

Black Duck Security Report

Merging #28 into develop will not change security risk.

Added Components

Clean: 2

Removed Components

High Risk: 1 Clean: 1

Click here to see full report