search

mesosphere / spark-build

Used to build the mesosphere/spark docker image and the DC/OS Spark package
52 stars 34 forks source link

Locally building fails #493

Closed wjglerum closed 5 years ago

wjglerum commented 5 years ago

I tried building this project locally to give spark 2.4.0 a try. I followed the README, however the Makefile doesn't work for me. Did I miss anything?

  1. make docker-build works and a container is created
  2. make prod-dist fails with: 
    /bin/bash: ./dev/make-distribution.sh: No such file or directory
make: *** [prod-dist] Error 127
akirillov commented 5 years ago

Hi @wjglerum, just to clarify prod-dist step is used to checkout Apache Spark sources and build a dist tarball for the future use with Dockerfile. If you're looking into creating a stub and deploy it to your DC/OS cluster, consider using publish_local_spark.sh script instead. Master branch of this repo depends on Spark 2.4 distribution.

Are you using a custom SPARK_DIR for the build? Is it an absolute path? Also, which version of make are you using?

If SPARK_DIRis not provided then mesosphere/spark master branch will be used which is a bit off of the upstream. You can try to checkout custom-branch-2.4.x which has all the latest changes and tested with all supported DC/OS versions.

So the whole process should look that:

mkdir dcos-spark 
cd dcos-spark
export DIR_PATH=$(pwd)

git clone git@github.com:mesosphere/spark.git
cd spark
git checkout custom-branch-2.4.x
cd ..

git clone git@github.com:mesosphere/spark-build.git
cd spark-build

SPARK_DIR=${DIR_PATH}/spark make prod-dist
wjglerum commented 5 years ago

Hi, thank you for your reply. I will try to test it again. For now Spark 2.4.0 is available already in the catalog so we are fine ;)