Build Spark from source

[razvan]

Description

:warning: CI: https://ci.stackable.tech/view/02%20Operator%20Tests%20(custom)/job/spark-k8s-operator-it-custom/132/

--- FAIL: kuttl (1905.79s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source_delta-3.1.0 (209.63s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source_s3-use-tls-true (156.61s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source_ny-tlc-report-0.1.0 (128.42s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (87.89s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source_ny-tlc-report-0.1.0 (284.48s)
        --- PASS: kuttl/harness/pod_overrides_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (114.80s)
        --- FAIL: kuttl/harness/iceberg_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (300.95s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (102.34s)
        --- PASS: kuttl/harness/spark-pi-public-s3_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (76.81s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (53.22s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source_s3-use-tls-true (205.42s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source_s3-use-tls-true (137.36s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-from-source (47.33s)
FAIL
ERROR:root:kuttl failed

Iceberg test successful too:

--- PASS: kuttl (63.62s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-3.5.1,docker.stackable.tech_sandbox_spark-k8s_3.5.1-stackable0.0.0-dev (63.60s)
PASS

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

- [ ] Changes are OpenShift compatible
- [ ] All added packages (via microdnf or otherwise) have a comment on why they are added
- [ ] Things not downloaded from Red Hat repositories should be mirrored in the Stackable repository and downloaded from there
- [ ] All packages should have (if available) signatures/hashes verified
- [ ] Does your change affect an SBOM? Make sure to update all SBOMs
- [ ] Add an entry to the CHANGELOG.md file
- [ ] Integration tests ran successfully

TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands: ```shell bake --product --image-version kind load docker-image --name= ``` See the output of `bake` to retrieve the image tag for ``.