search

FRosner / drunken-data-quality

Spark package for checking data quality
Apache License 2.0
222 stars 69 forks source link

pyddq #102

Closed Gerrrr closed 8 years ago

Gerrrr commented 8 years ago

This PR is a solution for https://github.com/FRosner/drunken-data-quality/issues/91

In order to test pyddq with Travis, please put ddq jar to python/drunken-data-quality.jar

As described in README.md, --driver-class-path cli option should be used instead of --packages because of https://issues.apache.org/jira/browse/SPARK-5185. I tried to make use of sc._jvm.java.lang.Thread.currentThread().getContextClassLoader().loadClass magic, but did not succeed.

FRosner commented 8 years ago

I am starting the review now, @Gerrrr. Are you going to fix the codacy complaints?

FRosner commented 8 years ago

@Gerrrr why are there commits in the merge request that add the gitter badge and the logo?

FRosner commented 8 years ago

Also, travis is not building because the branch is out-of-date with master. Please do not simply merge but try to find out why we have duplicate commits (you did something wrong at some point?). After the merge has been done and conflicts are resolved, travis can build again.

FRosner commented 8 years ago

@Gerrrr the python install command seems to fail. Do we need sudo rights?

FRosner commented 8 years ago

@Gerrrr do I remember correctly that I needed to provide a custom SBT task that creates drunken-data-quality.jar?

Gerrrr commented 8 years ago

@FRosner, yes it is expected that the build puts the jar to python/drunken-data-quality.jar.

FRosner commented 8 years ago

@Gerrrr right now, sbt assembly will also execute all tests on the Scala side. Is this ok? Or shall we disable the tests for assembly?

codecov-io commented 8 years ago

Current coverage is 100%

Merging #102 into master will not change coverage

@@            master   #102   diff @@
=====================================
  Files           24     24          
  Lines          437    437          
  Methods        429    421     -8   
  Messages         0      0          
  Branches         8     16     +8   
=====================================
  Hits           437    437          
  Misses           0      0          
  Partials         0      0

Powered by Codecov. Last updated by e2510c3...fef0d91

FRosner commented 8 years ago

I squashed your commits and opened https://github.com/FRosner/drunken-data-quality/pull/103. I will close this PR.