Create Spark Package - Githubissues

FRosner / drunken-data-quality

Spark package for checking data quality

Apache License 2.0

222 stars 69 forks source link

Create Spark Package #53

Closed FRosner closed 8 years ago

FRosner commented 8 years ago

Task

http://spark-packages.org/register

Acceptance Criteria

[x] Package is visible on http://spark-packages.org => http://spark-packages.org/package/FRosner/drunken-data-quality
[x] Package can be included via --package option when launching Spark (https://github.com/databricks/sbt-spark-package)
[x] Including DDQ as a dependency through spark packages sbt plugin works and is documented in README

FRosner commented 8 years ago

I opened https://github.com/databricks/sbt-spark-package/issues/15 because I could not add the sbt-spark-package SBT plugin.

FRosner commented 8 years ago

Fixed now. After it is merged I have to test it when I am at #59. I can close it once all acceptance criteria are fulfilled.

FRosner commented 8 years ago

Packages and releases submitted for verification:
type    name    user    create time updated time    state   message
release FRosner/drunken-data-quality:2.1.0  FRosner 2016-02-09 07:44:15 2016-02-09 07:52:03 error   u"pom file doesn't exist in zip archive or is misnamed. Was looking for drunken-data-quality-2.1.0.pom, contents: [u'drunken-data-quality-2.1.0-s_2.10.jar', u'drunken-data-quality-2.1.0-s_2.10.pom']"

FRosner commented 8 years ago

From the feedback service I got the reply:

Regarding supporting multiple Scala versions, when using the sbt-spark-package plugin, you may use +spDist. Note the plus sign. You will also need to set the settings for "crossScalaVersions" and "spAppendScalaVersion". You may check out the build file of spark-csv for reference.

Will give it a try now.