Issue with building images from resource-manager/kubernetes

Hi, I have performed the following steps

git clone https://github.com/apache-spark-on-k8s/spark.git git branch
- branch-2.2-kubernetes
build/mvn install -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
build/mvn compile -T 4C -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests

/spark$ docker build -t /spark-base:2.3.2 -f resource-managers/kubernetes/docker-minimal-bundle/src/main/docker/spark-base/Dockerfile . Sending build context to Docker daemon 362MB Step 1/10 : FROM openjdk:8-alpine ---> 97bc1352afde Step 2/10 : RUN apk upgrade --no-cache && apk add --no-cache bash tini && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && chgrp root /etc/passwd && chmod ug+rw /etc/passwd ---> Using cache ---> 3089c69ab995 Step 3/10 : COPY jars /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder893316008/jars: no such file or directory

Please note that I am able to do the build the docker image on spark-2.3.2-bin-hadoop2.7

docker build -t /spark-base:2.3.2 -f kubernetes/dockerfiles/spark/Dockerfile . Sending build context to Docker daemon 256.6MB Step 1/14 : FROM openjdk:8-alpine ---> 97bc1352afde Step 2/14 : ARG spark_jars=jars ---> Using cache ---> 80dfc6d3c51e Step 3/14 : ARG img_path=kubernetes/dockerfiles ---> Using cache ---> 548074284c13 Step 4/14 : RUN set -ex && apk upgrade --no-cache && apk add --no-cache bash tini libc6-compat && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && chgrp root /etc/passwd && chmod ug+rw /etc/passwd ---> Running in 4b80acdd4295

apk upgrade --no-cache fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz (1/1) Upgrading lcms2 (2.9-r0 -> 2.9-r1) OK: 101 MiB in 53 packages
apk add --no-cache bash tini libc6-compat fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz (1/7) Installing ncurses-terminfo-base (6.1_p20180818-r1) (2/7) Installing ncurses-terminfo (6.1_p20180818-r1) (3/7) Installing ncurses-libs (6.1_p20180818-r1) (4/7) Installing readline (7.0.003-r0) (5/7) Installing bash (4.4.19-r1) Executing bash-4.4.19-r1.post-install (6/7) Installing libc6-compat (1.1.19-r10) (7/7) Installing tini (0.18.0-r0) Executing busybox-1.28.4-r1.trigger OK: 109 MiB in 60 packages
mkdir -p /opt/spark
mkdir -p /opt/spark/work-dir touch /opt/spark/RELEASE
rm /bin/sh
ln -sv /bin/bash /bin/sh '/bin/sh' -> '/bin/bash'
chgrp root /etc/passwd
chmod ug+rw /etc/passwd Removing intermediate container 4b80acdd4295 ---> 2544b26afc10 Step 5/14 : COPY ${spark_jars} /opt/spark/jars ---> 6347395c7ae4 Step 6/14 : COPY bin /opt/spark/bin ---> fd24ba7fc215 Step 7/14 : COPY sbin /opt/spark/sbin ---> 2bb05d005262 Step 8/14 : COPY conf /opt/spark/conf ---> 66862a50640b Step 9/14 : COPY ${img_path}/spark/entrypoint.sh /opt/ ---> f2540df39d35 Step 10/14 : COPY examples /opt/spark/examples ---> b26bfb356664 Step 11/14 : COPY data /opt/spark/data ---> 2a472cf28f86 Step 12/14 : ENV SPARK_HOME /opt/spark ---> Running in 1e4abc5ac922 Removing intermediate container 1e4abc5ac922 ---> 5baa2d4ad0dc Step 13/14 : WORKDIR /opt/spark/work-dir Removing intermediate container 6c55a299f8c4 ---> df0219347271 Step 14/14 : ENTRYPOINT [ "/opt/entrypoint.sh" ] ---> Running in cb13ea338f8e Removing intermediate container cb13ea338f8e ---> ccaacc12774a Successfully built ccaacc12774a Successfully tagged /spark-base:2.3.2

Is this issue specific to this branch?

This fork is deprecated and you should be using upstream Spark master instead

As for your specific issue I expect your mvn arguments are insufficient to create all the artifacts that the Docker build wants to package. Your build context looks far too small (362MB) compared to what I normally see when building from upstream Spark master (~1.2GB) so you certainly have some artifacts missing.

Personally I would normally do a mvn package with the relevant -P profiles but without the -pl arguments

@rvesse Thanks. I did the following at your advice /spark$ git branch

master

build/mvn -DskipTests clean package [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 7.388 s] [INFO] Spark Project Tags ................................. SUCCESS [ 32.795 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 8.640 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 13.247 s] [INFO] Spark Project Networking ........................... SUCCESS [ 17.010 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.552 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 12.738 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 18.440 s] [INFO] Spark Project Core ................................. SUCCESS [05:04 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 40.713 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 23.278 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 50.816 s] [INFO] Spark Project Catalyst ............................. SUCCESS [02:46 min] [INFO] Spark Project SQL .................................. SUCCESS [04:08 min] [INFO] Spark Project ML Library ........................... SUCCESS [02:23 min] [INFO] Spark Project Tools ................................ SUCCESS [ 2.878 s] [INFO] Spark Project Hive ................................. SUCCESS [01:10 min] [INFO] Spark Project REPL ................................. SUCCESS [ 11.779 s] [INFO] Spark Project Assembly ............................. SUCCESS [ 6.403 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 14.769 s] [INFO] Kafka 0.10 Source for Structured Streaming ......... SUCCESS [ 18.781 s] [INFO] Spark Project Examples ............................. SUCCESS [ 25.863 s] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 5.858 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 20:54 min [INFO] Finished at: 2018-11-08T14:43:57+01:00 [INFO] Final Memory: 74M/327M [INFO] ------------------------------------------------------------------------

build/mvn install -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests .... .... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 13.573 s] [INFO] Spark Project Tags ................................. SUCCESS [ 4.918 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 3.321 s] [INFO] Spark Project Networking ........................... SUCCESS [ 6.795 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.378 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 2.694 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 7.443 s] [INFO] Spark Project Core ................................. SUCCESS [01:01 min] [INFO] Spark Project Kubernetes ........................... SUCCESS [ 14.983 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:00 min [INFO] Finished at: 2018-11-08T14:48:07+01:00 [INFO] Final Memory: 63M/276M [INFO] ------------------------------------------------------------------------

/spark$ docker build -t aarthipa/spark-kubernetes:2.3.2 -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . Sending build context to Docker daemon 1.308GB Step 1/13 : FROM openjdk:8-alpine ---> 97bc1352afde Step 2/13 : ARG spark_jars=jars ---> Using cache ---> 80dfc6d3c51e Step 3/13 : ARG img_path=kubernetes/dockerfiles ---> Using cache ---> 548074284c13 Step 4/13 : RUN set -ex && apk upgrade --no-cache && apk add --no-cache bash tini libc6-compat && mkdir -p /opt/spark && mkdir -p /opt/spark/work-dir touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && chgrp root /etc/passwd && chmod ug+rw /etc/passwd ---> Using cache ---> 2544b26afc10 Step 5/13 : COPY ${spark_jars} /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder253012913/jars: no such file or directory

Still the same error though :(

You can try customising the Docker build arguments to set spark_jars correctly but you failed to take my main advice which is that this fork is defunct!

Please stop using it and switch to upstream Spark (either branch-2.4 or master) which are actively maintained.

Thanks very much @rvesse . Im now using the master branch of https://github.com/apache/spark.git which I suppose is the upstream branch. I still have the same issue with spark_jars? What exactly are the spark_jars here? Does it refer to the spark core jars or my application jar?

The following are the build results build/mvn -DskipTests clean package [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM 3.0.0-SNAPSHOT ............ SUCCESS [ 27.249 s] [INFO] Spark Project Tags ................................. SUCCESS [ 46.819 s] [INFO] Spark Project Sketch ............................... SUCCESS [ 10.992 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 10.410 s] [INFO] Spark Project Networking ........................... SUCCESS [ 19.719 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 9.258 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 13.617 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 38.186 s] [INFO] Spark Project Core ................................. SUCCESS [05:33 min] [INFO] Spark Project ML Local Library ..................... SUCCESS [ 14.475 s] [INFO] Spark Project GraphX ............................... SUCCESS [ 25.300 s] [INFO] Spark Project Streaming ............................ SUCCESS [ 52.231 s] [INFO] Spark Project Catalyst ............................. SUCCESS [02:44 min] [INFO] Spark Project SQL .................................. SUCCESS [05:12 min] [INFO] Spark Project ML Library ........................... SUCCESS [02:59 min] [INFO] Spark Project Tools ................................ SUCCESS [ 4.123 s] [INFO] Spark Project Hive ................................. SUCCESS [01:06 min] [INFO] Spark Project REPL ................................. SUCCESS [ 11.096 s] [INFO] Spark Project Assembly ............................. SUCCESS [ 9.185 s] [INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 20.578 s] [INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 24.695 s] [INFO] Spark Project Examples ............................. SUCCESS [ 27.765 s] [INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 9.225 s] [INFO] Spark Avro 3.0.0-SNAPSHOT .......................... SUCCESS [ 10.002 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 24:02 min [INFO] Finished at: 2018-11-09T10:14:23+01:00 [INFO] ------------------------------------------------------------------------

build/mvn install -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM 3.0.0-SNAPSHOT ............ SUCCESS [ 10.692 s] [INFO] Spark Project Tags ................................. SUCCESS [ 5.836 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 4.335 s] [INFO] Spark Project Networking ........................... SUCCESS [ 8.872 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 3.920 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 2.822 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 7.456 s] [INFO] Spark Project Core ................................. SUCCESS [01:03 min] [INFO] Spark Project Kubernetes 3.0.0-SNAPSHOT ............ SUCCESS [01:03 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:52 min [INFO] Finished at: 2018-11-09T10:18:22+01:00 [INFO] ------------------------------------------------------------------------

docker build -t aarthipa/spark-kubernetes:2.3.2 -f resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile . Sending build context to Docker daemon 1.387GB Step 1/17 : FROM openjdk:8-alpine ---> 97bc1352afde Step 2/17 : ARG spark_jars=jars ---> Using cache ---> 80dfc6d3c51e Step 3/17 : ARG example_jars=examples/jars ---> Running in 59fdcdee16b8 Removing intermediate container 59fdcdee16b8 ---> 664c89887c35 Step 4/17 : ARG img_path=kubernetes/dockerfiles ---> Running in 590f78ec61b3 Removing intermediate container 590f78ec61b3 ---> 1a0ca49a971b Step 5/17 : ARG k8s_tests=kubernetes/tests ---> Running in 261cb67e2a6e Removing intermediate container 261cb67e2a6e ---> fd73de35bebd Step 6/17 : RUN set -ex && apk upgrade --no-cache && apk add --no-cache bash tini libc6-compat linux-pam krb5 krb5-libs && mkdir -p /opt/spark && mkdir -p /opt/spark/examples && mkdir -p /opt/spark/work-dir && touch /opt/spark/RELEASE && rm /bin/sh && ln -sv /bin/bash /bin/sh && echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && chgrp root /etc/passwd && chmod ug+rw /etc/passwd ---> Running in 3984d497684b

apk upgrade --no-cache fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz (1/1) Upgrading lcms2 (2.9-r0 -> 2.9-r1) OK: 101 MiB in 53 packages
apk add --no-cache bash tini libc6-compat linux-pam krb5 krb5-libs fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz (1/10) Installing ncurses-terminfo-base (6.1_p20180818-r1) (2/10) Installing ncurses-terminfo (6.1_p20180818-r1) (3/10) Installing ncurses-libs (6.1_p20180818-r1) (4/10) Installing readline (7.0.003-r0) (5/10) Installing bash (4.4.19-r1) Executing bash-4.4.19-r1.post-install (6/10) Installing e2fsprogs-libs (1.44.2-r0) (7/10) Installing krb5 (1.15.3-r0) (8/10) Installing libc6-compat (1.1.19-r10) (9/10) Installing linux-pam (1.3.0-r0) (10/10) Installing tini (0.18.0-r0) Executing busybox-1.28.4-r1.trigger OK: 111 MiB in 63 packages
mkdir -p /opt/spark
mkdir -p /opt/spark/examples
mkdir -p /opt/spark/work-dir
touch /opt/spark/RELEASE
rm /bin/sh
ln -sv /bin/bash /bin/sh '/bin/sh' -> '/bin/bash'
echo 'auth required pam_wheel.so use_uid'
chgrp root /etc/passwd
chmod ug+rw /etc/passwd Removing intermediate container 3984d497684b ---> 84bc2fb07514 Step 7/17 : COPY ${spark_jars} /opt/spark/jars COPY failed: stat /var/lib/docker/tmp/docker-builder208101562/jars: no such file or directory

spark_jars is a build argument passed to the docker build command via the --build-arg flag.

If you are using Apache Spark there is a convenience script bin/docker-image-tool.sh which handles the logic of setting the build arguments appropriately based upon your build environment e.g.

> bin/docker-image-tool.sh -r <repo> -t <tag> build

See the official documentation at https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images

Thanks again. Its all working now :)

apache-spark-on-k8s / spark

Issue with building images from resource-manager/kubernetes #634