added sbt-spark-package support

[brkyvz]

Hi!

I added support for the sbt-spark-package plugin with this PR. With this plugin, you can make releases to Spark Packages directly using sbt! No need to go to the website anymore. All you need to do is (since you publish to Maven Central as well) to publish to Maven Central first, using sbt publish. Then you can publish to Spark Packages using sbt spPublish.

The one thing to be careful about is that the HEAD commit on your local repo must exist on your remote repository as well, so that we can pull in your tree from Github during checks. In addition, one last step is to add a Github personal access token into a credentials file as described in the README.

We would love to see your package on the Spark Packages repository as well! (Also, your package will be ranked higher on the website :))

Best, Burak

[koertkuipers]

Thanks Burak! Does the plugin simply register the release (so the same as when i log into spark-packages and register a release manually), or does it do more than that?

[brkyvz]

It has a lot of features:

• sbt console spawns up spark-shell for easy testing during development
• sbt spRegister registers a new package
• sbt spPublish publishes the release as you would register a release on the website
• sbt spPackage packages a jar (including python files if you have any so that it can work on pyspark)
• sbt assembly assembles an uber jar including python files (if you have any)

basically it gives you a lot of functionality to test and develop spark packages.