Submit fortis package to maven

[jcjimenez]

It looks like the artifacts for fortis-spark (and possibly others) aren't making it into maven. Definition of done: all artifacts required for one-click deploy should be published to maven.

[c-w]

Note that the current way of publishing packages as demonstrated by packages like streaming-instagram won't work for project-fortis-spark as we need to publish a fat jar. Basically, we need to tell sbt-release to use the output of sbt-assembly and not the default output of sbt-package.

Here is a stackoverflow post that describes how we can achieve this or here the sbt-assembly docs.

[c-w]

Actually, this is already implemented in our sbt file so we should just need to follow the standard publishing steps:

sbt
sonatypeOpen "0.0.1"
publishSigned
sonatypeRelease

[c-w]

Running through the steps now. For some reason I got an error stating gpg: can't connect to the agent: IPC connect call failed. Fix: manually start the gpg agent in a separate shell: "C:\Program Files (x86)\GNU\GnuPG\gpg-connect-agent.exe" --verbose

[c-w]

[info] SHA-1: 26c3a60509d896f23003fd741edb7c51970fef69
[info] Packaging C:\Repos\project-fortis-spark\target\scala-2.11\project-fortis-spark-assembly-0.0.1.jar ...
[info] Done packaging.
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1-javadoc.jar.asc
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1-sources.jar
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1.pom.asc
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1.pom
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1-assembly.jar
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1.jar
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1-assembly.jar.asc
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1-javadoc.jar
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1.jar.asc
[info]  published project-fortis-spark_2.11 to https://oss.sonatype.org/service/local/staging/deploy/maven2/default/project-fortis-spark_2.11/0.0.1/project-fortis-spark_2.11-0.0.1-sources.jar.asc
[success] Total time: 302 s, completed Jun 27, 2017 8:22:58 AM

[kevinhartman]

I'd added publishing support a while back but did not publish since our code was mostly a demo (and still is not really V1). We should be able to overwrite this in Ivy in the beginning stages.

[c-w]

I keep getting errors from sonatype that it doesn't find staging profiles. Could someone else try this?

[c-w]

Example setup on ibex-dashboard for publishing from Travis: .travis.yml, push.sh. The difference to our pipeline is that ibex uses github as the publishing target so we just need to push commits as opposed to publish to an external service.

To publish from Travis to Maven, we'll need a few things: 1) Add environment variables SONATYPE_USER and SONATYPE_PASSWORD to the Travis build via settings 2) After build success, check if we're building a tag by looking at TRAVIS_TAG 3) If so: run the sbt publishing commands

Sample addition to the Travis build file:

after_success:
  - if [ ! -z "${TRAVIS_TAG}" ]; then sbt 'sonatypeOpen "travis-${TRAVIS_TAG}"'; sbt publishSigned; sbt sonatypeRelease; fi

Open questions: a) How do we provide the credentials for GPG to sign the binaries?

[c-w]

Getting GPG to sign the jars means deploying a trusted key to Travis. This would require us uploading the key somewhere like Azure Blob, securing that location with an access key and then downloading the key to the Travis machine (with the access key provided via a secure environment variable). This is somewhat complex. It would be easier to just upload the built far jar to a known location like Azure Blob and then spark-submit it from there.

[c-w]

There already exists a way to natively upload the gpg key to Travis. However, publishing the fat jar via Maven will still require Travis to impersonate one of our GPG keys which is not really what the system was designed for. The more I think about this, the more I think that we should just deploy the fat jar via blob. The main benefit here is that we're not dependent on the Maven sync for redeploying jar changes. We've already seen that it can take a while for published jars to show up in Maven (half a day sometimes) which may be annoying when we have to deploy a hotfix to the pipeline. If we go with the Travis/Blob solution, we can re-deploy new versions instantly.