Upgrade dependency versions, remove legacy dependencies and fix travis build errors

FINRAOS / MegaSparkDiff

A Spark-based data comparison tool at scale which facilitates software development engineers to compare a plethora of pair combinations of possible data sources. Multiple execution modes in multiple environments enable the user to generate a diff report as a Java/Scala-friendly DataFrame or as a file for future use. Comes with out of the box SparkFactory and SparkCompare tools.

https://finraos.github.io/MegaSparkDiff/

Apache License 2.0

48 stars 26 forks source link

Upgrade dependency versions, remove legacy dependencies and fix travis build errors #57

Closed nrthyrk closed 4 years ago

nrthyrk commented 5 years ago

upgrade dependency versions
- upgraded spark version from 2.3.1 to 2.4.4, to conform with the latest version of EMR. We should keep spark version with the latest.
- upgraded scala version to 2.11.12
- upgraded other dependencies to latest
cleanup legacy dependencies
- removed commons-lang3, as it's not in use
- removed commons-codec, as it's not in use; and the 1.11 version is conflicting with hadoop library dependencies latest EMR version is using. The comment in our pom.xml says "Used for base-64 encoding for HTTP basic authentication (which we need to access Jenkins)" for this dependency, but I did not find such a usage.
travis build errors
- there were jdk version compatibility issue, added jdk version in travis yml file.

wnilkamal commented 5 years ago

Migration Recommended

It appears you are still using the CoPilot OAuth App - we recommend you migrate to the GitHub App

wnilkamal commented 5 years ago

Migration Recommended

It appears you are still using the CoPilot OAuth App - we recommend you migrate to the GitHub App

nrthyrk commented 5 years ago

Migration Recommended

It appears you are still using the CoPilot OAuth App - we recommend you migrate to the GitHub App

how do i do that?

matthewgillett commented 5 years ago

Closing and reopening pull request to re-trigger checks.

blackduck-copilot[bot] commented 5 years ago

Black Duck Security Report

Merging #57 into develop will not change security risk.

Added Components

Medium Risk: 1 Clean: 53

Removed Components

High Risk: 2 Clean: 54

Click here to see full report

5 component(s) did not change but have updated security data. This may affect your calculated score.