Simple script to sinchronize spark mainline with spark on nomad fork

[tantra35]

Due this fork not actively maintained, we want to contribute some of our work:

This simple script allow to quickly synchronize spark on nomad with minor spark mainline

#!/bin/bash

SPARKVERSION=2.3.4
SPARKVERSIONDIFF=-2 #SPARKONNOMADVERSION
SPARKBUILDDIR=/home/vagrant/build/spark-on-nomad-${SPARKVERSION}

# https://stackoverflow.com/questions/8653126/how-to-increment-version-number-in-a-shell-script
increment_version ()
{
  declare -a part=( ${1//\./ } )
  declare    new
  declare -i carry=$2

  for (( CNTR=${#part[@]}-1; CNTR>=0; CNTR-=1 )); do
    len=${#part[CNTR]}
    new=$((part[CNTR]+carry))
    [ ${#new} -gt $len ] && carry=1 || carry=0
    [ $CNTR -gt 0 ] && part[CNTR]=${new: -len} || part[CNTR]=${new}
  done
  new="${part[*]}"
  echo -e "${new// /.}"
}

if [ -z $SPARKONNOMADVERSION ]; then
    SPARKONNOMADVERSION=`increment_version $SPARKVERSION $SPARKVERSIONDIFF`
fi

echo Original spark: $SPARKVERSION, patches will by applyed from $SPARKONNOMADVERSION nomad spark fork

if [ -e $SPARKBUILDDIR ]; then
    echo  "Removing prev build dir"
    rm -fr $SPARKBUILDDIR
fi

git config --global user.email "support@playrix.com"
git config --global user.name "Playrix LLC"

mkdir -p /home/vagrant/build
git clone https://github.com/hashicorp/nomad-spark.git ${SPARKBUILDDIR}

cd ${SPARKBUILDDIR}
git remote add upstream https://github.com/apache/spark.git
git fetch upstream
git checkout origin/nomad-on-spark-${SPARKONNOMADVERSION}
git checkout -b nomad-on-spark-${SPARKVERSION}
git merge -m "merge with mainlain spark ${SPARKVERSION}" tags/v${SPARKVERSION}

cp /home/vagrant/build/spark-on-nomad/build.sh ${SPARKBUILDDIR}
chmod 0755 ${SPARKBUILDDIR}/build.sh

sed -ri "s!${SPARKONNOMADVERSION}!${SPARKVERSION}!" ${SPARKBUILDDIR}/resource-managers/nomad/pom.xml
sed -ri "s!${SPARKONNOMADVERSION}!${SPARKVERSION}!" ${SPARKBUILDDIR}/resource-managers/nomad/test-apps/pom.xml

sed -ri "s!0.8.6.1!0.7.0.2!" ${SPARKBUILDDIR}/resource-managers/nomad/pom.xml
sed -ri "s!0.8.6.1!0.7.0.2!" ${SPARKBUILDDIR}/resource-managers/nomad/test-apps/pom.xml

${SPARKBUILDDIR}/build.sh
cp ${SPARKBUILDDIR}/spark-${SPARKVERSION}-bin-nohadoop-nomad.tgz /home/vagrant/build/spark-on-nomad/

and this allow to build it build.sh (without inline hadoop, we use separate hadoop installation of version 2.9.2):

#!/bin/bash

./dev/make-distribution.sh --name nohadoop-nomad --tgz -Pmesos -Pnomad -Psparkr -Phive -Phadoop-2.7 -Phadoop-provided -Phive-thriftserver -DskipTests
#./build/mvn -Pmesos -Pnomad -Psparkr -Phive -Phadoop-2.7 -Phive-thriftserver -DskipTests clean package 

and here is our Vagrant file

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
    config.vm.box = "ubuntu/xenial64"

    config.vm.define "build-spark-on-nomad" do |node|
        node.vm.hostname = "build-spark-on-nomad"
        node.vm.synced_folder "./", "/home/vagrant/build/spark-on-nomad/"

        node.vm.provider "virtualbox" do |vb|
            vb.memory = "4096"
        end

        node.vm.provision "shell", inline: <<-SHELL
            add-apt-repository ppa:webupd8team/java -y
            echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
            apt-get update
            apt-get install -y oracle-java8-installer
            apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
            add-apt-repository 'deb [arch=amd64,i386] https://cran.rstudio.com/bin/linux/ubuntu xenial/'
            apt-get update
            apt-get install -y r-base
            apt-get install -y git
            apt-get install -y mc
            ln -s "$(which python3)" /usr/local/python
        SHELL
    end
end

[OneCricketeer]

Status?

[tantra35]

@cricket007 what status you are asking about?

[OneCricketeer]

No one's responded to this in almost a year...