search

gettyimages / docker-spark

Docker build for Apache Spark
MIT License
676 stars 370 forks source link

Docker image with tag 2.4.0 actually is 2.3.1 #51

Closed fubhy closed 5 years ago

fubhy commented 5 years ago

The published Docker image with tag 2.4.0-hadoop-3.0 actually contains spark version 2.3.1. However, the latest tag has 2.4.0.

cclient commented 5 years ago 
FROM gettyimages/spark:2.4.0-hadoop-3.0
ENV SPARK_VERSION 2.4.0
ENV SPARK_HOME /usr/spark-${SPARK_VERSION}
ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/jdk1.8.0_192/bin:/usr/hadoop-3.0.0/bin:/usr/spark-2.4.0/bin:
ADD spark-2.4.0 /usr/spark-2.4.0
RUN chown -R root:root /usr/spark-2.4.0
WORKDIR /usr/spark-2.4.0

docker build -t spark:2.4.0-hadoop-3.0 ./

build by self for temporary use (1.23GB)

docker pull cuidapeng:spark:2.4.0-hadoop-3.0

https://cloud.docker.com/u/cuidapeng/repository/docker/cuidapeng/spark

calba commented 5 years ago

I am not a big fan of using latest but published image with latest tag has actually 2.4.0

bryceageno commented 5 years ago

sorry about this. I pushed the tag with the upgrade so it built with the old code.

calba commented 5 years ago

Thanks for response.

Unfortunately, it seems the build broke. In case it is easier. Is it possible to just retag the current latest as 2.4.0-hadoop-3.0?

bryceageno commented 5 years ago

Yes I removed then added back the tag so it now points to latest, I am working on getting permissions to the docker hub repository so I can delete the old version and repush.

bryceageno commented 5 years ago

This is now fixed, it was failing due to a removed java version