awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
635 stars 299 forks source link

mvn build cannot find `tools.jar` #177

Open arnold-jr opened 1 year ago

arnold-jr commented 1 year ago

I ran into an error when building the v4.0 release. It seems that a dependency is trying to use a relative path to load ../lib/tools.jar. This can be resolved by copying the jar to the location it expects, but I've noticed this jar is not present in the openjdk distro.

FROM amazoncorretto:8-alpine-jdk

# Copies the aws-glue-libs source
WORKDIR /opt
RUN wget https://github.com/awslabs/aws-glue-libs/archive/refs/tags/v4.0.tar.gz
RUN tar xzf v4.0.tar.gz && rm v4.0.tar.gz && ln -s aws-glue-libs-4.0 aws-glue-libs

# Downloads AWS-specific maven
RUN wget https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
RUN tar xvzf apache-maven*.tar.gz && rm apache-maven*.tar.gz
ENV PATH=/opt/apache-maven-3.6.0/bin:$PATH

WORKDIR /opt/aws-glue-libs
RUN  mvn -f ./pom.xml -DoutputDirectory=./jarsv1 dependency:copy-dependencies
# -- This step FAILS: 
# [ERROR] Failed to execute goal on project AWSGlueETLPython: Could not resolve dependencies for project com.amazonaws:AWSGlueETLPython:jar:4.0.0: Could not find artifact jdk.tools:jdk.tools:jar:1.8 at specified path /usr/lib/jvm/java-8-amazon-corretto/../lib/tools.jar -> [Help 1]

Here's the hack:

FROM amazoncorretto:8-alpine-jdk

ENV SPARK_HOME=/opt/spark
ENV SPARK_CONF_DIR=${SPARK_HOME}/conf
ENV PYSPARK_PYTHON_DRIVER=python3

# Copies the aws-glue-libs source
WORKDIR /opt
RUN wget https://github.com/awslabs/aws-glue-libs/archive/refs/tags/v4.0.tar.gz
RUN tar xzf v4.0.tar.gz && rm v4.0.tar.gz && ln -s aws-glue-libs-4.0 aws-glue-libs

# Downloads AWS-specific maven
RUN wget https://aws-glue-etl-artifacts.s3.amazonaws.com/glue-common/apache-maven-3.6.0-bin.tar.gz
RUN tar xvzf apache-maven*.tar.gz && rm apache-maven*.tar.gz
ENV PATH=/opt/apache-maven-3.6.0/bin:$PATH

WORKDIR /usr/lib/jvm
RUN mkdir lib && cp /usr/lib/jvm/default-jvm/lib/tools.jar lib/tools.jar 

WORKDIR /opt/aws-glue-libs
RUN  mvn -f ./pom.xml -DoutputDirectory=./jarsv1 dependency:copy-dependencies