search

mesosphere / spark-build

Used to build the mesosphere/spark docker image and the DC/OS Spark package
52 stars 34 forks source link

[DCOS-54709] Scala 2.12 support #536

Closed rpalaznik closed 5 years ago

rpalaznik commented 5 years ago

What changes were proposed in this pull request?

Resolves DCOS-54709 - Add Scala 2.12 support for Spark

How were these changes tested?

I wanted to avoid creating test tags in git, so I tested the changes manually without using any production credentials, except uploading new spark distributions to downloads.mesosphere.io domain.

1) Run the following script to build spark distribution archives:

SPARK_VERSION=2.4.3 \
DIST_DESTINATION_DIR=/tmp \
bin/jenkins-dist-publish.sh

Executed in spark-tools container, as described in the wiki. I had mesosphere/spark and mesosphere/spark-build repositories checked out and mounted next to each other. I've added DIST_DESTINATION_DIR variable to skip upload to S3 and save the archives locally.

2) Uploaded the spark distributions to downloads.mesosphere.io and updated manifest.json

3) Run the following script to build and push docker images to my personal docker hub account

DOCKER_USERNAME=rpalaznik \
DOCKER_PASSWORD=[redacted :)] \
SPARK_BUILD_VERSION=2.4.3 \
DOCKER_DIST_IMAGE=rpalaznik/spark \
GITHUB_DISABLE=true \
bin/jenkins-package-publish.sh

As I mentioned, this is just to test that the script is working. When making an actual release, you should simply create a release tag and run the jenkins job as described in the wiki.

4) Run the smoke test for custom docker images using the instructions from the wiki on how to run integration tests and by passing these variables

If TEST_SH_SMOKE_TEST_SPARK_DOCKER_IMAGES is not set, e.g. during a CI integration test run, the smoke test will be skipped.

Release Notes