[Feature] Base spark image build is very slow and impacting ci/cd

[daw3rd]

Search before asking

• [X] I searched the issues and found no similar issues.

Component

Library/core

Feature

While working, the build of the base spark image used by transforms enabled in the spark runtime, is very slow - about 20 minutes in ci/cd builds. With the recent change on ci/cd to run each transform separately, this means that each spark-based transform is doing this. Not too bad when only a transform is changed, but when the core library is changed, all transforms are built so that this is problem is even more painful.

It seems to be happening on this step

RUN wget https://archive.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_MAJOR_VERSION}.tgz && \
    tar xf spark-${SPARK_VERSION}-bin-hadoop${HADOOP_MAJOR_VERSION}.tgz -C /opt && \
    mv /opt/spark-${SPARK_VERSION}-bin-hadoop${HADOOP_MAJOR_VERSION} /opt/spark

ans specifically the wget.

To build the image

cd data-processing-lib/spark
make image

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

[daw3rd]

We may have discussed, but using a base spark image would probably help this.

[cmadam]

Why is the base image rebuilt every time a transform image is built? By default, the transform image uses a base image from a docker registry. That does not require rebuilding the base image for every CI/CD run. It sounds like something needs to be changed in the CI/CD pipeline itself.

[blublinsky]

++++1 to @cmadam