search

databricks / sbt-spark-package

Sbt plugin for Spark packages
Apache License 2.0
151 stars 32 forks source link

Local shared repository #2

Closed huangjs closed 9 years ago

huangjs commented 9 years ago

How can I setup a shared repository for spark packages, for managing non-open-sourced packages we build internally?

Jianshi

brkyvz commented 9 years ago

Hi, I'm not sure if I understood your question correctly, but I'll give it a try. The Spark Packages repository is a maven repository on Bintray. You can set up a private repo on Bintray, or a host of your choice and add that repository as a resolver in your build.sbt (as we do to add the Spark Package repo here ).

Burak

huangjs commented 9 years ago

I see. Then i can just publish to our local nexus. Just wondering how it would package python code for distribution.

Jianshi

brkyvz commented 9 years ago

spPackage packages the python code in the root directory of the jar (in addition to the Java/Scala class files). When the jar is added to the sys.path or PYTHONPATH, python codes should run without a problem.

Burak

huangjs commented 9 years ago

Ah it's in the jar. I see. Thanks for the explanation. Would you recommend using it for all Spark apps? As it handles python by default which saves people some configurations. :)

Jianshi

On Sun, Mar 15, 2015 at 3:43 AM, Burak Yavuz notifications@github.com wrote:

spPackage packages the python code in the root directory of the jar (in addition to the Java/Scala class files). When the jar is added to the sys.path or PYTHONPATH, python codes should run without a problem.

Burak

— Reply to this email directly or view it on GitHub https://github.com/databricks/sbt-spark-package/issues/2#issuecomment-80690646 .

Jianshi Huang

LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/

brkyvz commented 9 years ago

Most certainly!

Burak