Feature: JDK 17 Upgrade Support Series - Spark modules/pipelines

cwoods-cpointe commented 2 months ago

Description

In https://github.com/boozallen/aissemble/issues/133 we modified the build-parent to use JDK 17. When building aiSSEMBLE with the updated build-parent, some modules do not build successfully. This is a series of ticket to fix broken modules. This issue will focus on the spark pipeline functionality

DOD

[x] The builds for following modules work:
- [x] extensions-data-delivery
- [x] extensions-messaging-kafka
- [x] Docker modules
- [x] aissemble-spark
- [x] aissemble-spark-infrastructure
- [x] aissemble-spark-operator
- [x] Helm modules
- [x] aissemble-spark-operator-chart
- [x] aissemble-spark-application-chart
- [x] extensions-helm-spark-infrastructure
[x] The MDA templates for the spark pipelines are working
[x] Write migrations for common issues. These changes need to be weighed on if the destructiveness is doable in downstream projects. These migrations should be in a new group
[x] Spark docker images are build on JDK 17 or later
- [x] update any base images that are not using JDK 17
Follow on: Spark and Pyspark pipelines can be executed to completion
Follow on: The Kafka messaging module builds and functions (within the pipeline step coordination)
Follow on: The spark persist types are testes and work: hive and delta-lake
Potential work to be split out of this ticket
- Follow on: File store functionality
- Follow on: pipeline step configuration

Test Strategy/Script

Build aiSSEMBLE java upgrade feature branch

Note that if the non updated modules are failing you can build just the spark modules with:

mvn clean install -pl :build-parent,:aissemble-quarkus-bom,:bom-component,:aissemble-foundation-core-python,:extensions-data-delivery,:aissemble-extensions-data-delivery-spark-py,:extensions-data-delivery-spark,:extensions-data-delivery-spark-neo4j,:extensions-data-delivery-spark-postgres,:aissemble-spark,:aissemble-spark-infrastructure,:aissemble-spark-operator,:aissemble-spark-application-chart,:aissemble-spark-operator-chart,:extensions-helm-spark-infrastructure,:aissemble-hive-metastore-service-chart,:aissemble-spark-history-chart,:aissemble-thrift-server-chart -am

Generate a new downstream project using 1.9.0-SNPASHOT

mvn archetype:generate -B -DarchetypeGroupId=com.boozallen.aissemble \
                      -DarchetypeArtifactId=foundation-archetype \
                      -DarchetypeVersion=1.10.0-SNAPSHOT \
                      -DartifactId=test-project\
                      -DgroupId=org.test \
                      -DprojectName='Test' \
                      -DprojectGitUrl=test.org/test-project\
&& cd test-project

Create 2 new pipeline MDAs. One spark and one pyspark
- 356-helper.zip
Run mvn clean install and resolve all manual actions relating to spark
- Spark operator, spark application, spark-infrastructure, kafka, s3-local

Update the pyspark pyproject.toml to point to the local path for foundation-core-python and extensions-data-delivery-spark-py. This is because we are on a feature branch and the artifacts are not being deployed

aissemble-foundation-core-python = {path = "/your/path/to/aissemble/foundation/aissemble-foundation-core-python", develop =true}
aissemble-extensions-data-delivery-spark-py = {path = "/your/path/to/aissemble/extensions/extensions-data-delivery/aissemble-extensions-data-delivery-spark-py", develop = true}
aissemble-foundation-pdp-client-python = {path = "/your/path/to/aissemble/foundation/foundation-security/aissemble-foundation-pdp-client-python", develop = true}
aissemble-foundation-encryption-policy-python = {path = "/your/path/to/aissemble/foundation/foundation-encryption/aissemble-foundation-encryption-policy-python", develop = true}
aissemble-extensions-encryption-vault-python = {path = "/your/path/to/aissemble/extensions/extensions-encryption/aissemble-extensions-encryption-vault-python", develop = true}

Update the spark-operator and spark-infrastructure helm chart dependency to the locally build charts. This is because we are on a feature branch and the artifacts are not being deployed
Deploy with tilt up
Verify all the spark related pods come up successfully
Note the pipeline are currently failing and will be part of the follow on

Test 2 - spark migration

Create a 1.9.1 downstream project

mvn archetype:generate -B -DarchetypeGroupId=com.boozallen.aissemble \
                      -DarchetypeArtifactId=foundation-archetype \
                      -DarchetypeVersion=1.9.2 \
                      -DartifactId=test-project\
                      -DgroupId=org.test \
                      -DprojectName='Test' \
                      -DprojectGitUrl=test.org/test-project\
&& cd test-project

Create 2 new pipeline MDAs. One spark and one pyspark
- 356-helper.zip
Build the project and add the 2 pipelines into the pom modules to be built

Once the pipelines are generated update the sparkConf to include the old and new valies

In the base-values.yaml add:

sparkApp:
spec:
sparkConf:
spark.yarn.executor.failuresValidityInterval: "2h"
spark.yarn.max.executor.failures: 10

In the dev-values.yaml add

sparkApp:
spec:
sparkConf:
spark.executor.failuresValidityInterval: "2h"
spark.executor.maxNumFailures: 10

Update the aissemble version to 1.10.0-SNAPSHOT
Perform the baton migrations with mvn org.technologybrewery.baton:baton-maven-plugin:baton-migrate
Verify the files base-values.yaml configs were updated to match the dev-values.yaml
Verify the dev-values.yaml files were not modified

References/Additional Context

None

cwoods-cpointe commented 1 month ago

Ran into a test issue with Spark needing to use an internal API that has been made inaccessible by default:

[ERROR]   Run 1: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x68999068) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x68999068

Resolved by exporting it in the surefire plugin <argLine>--add-exports java.base/sun.nio.ch=ALL-UNNAMED</argLine>

cwoods-cpointe commented 1 month ago

spark/java support matrix shows 3.4.x supports java17 so docker images should be good - link

cwoods-cpointe commented 1 month ago

There is a conflict with our jackson and spark dependencies Scala module 2.14.2 requires Jackson Databind version >= 2.14.0 and < 2.15.0 - Found jackson-databind version 2.15.0. To mitigate this, we should try and bump up our spark version

colinpalmer-pro commented 1 month ago

Test Steps passed. Ticket Resolved.

boozallen / aissemble