Booz Allen's lean manufacturing approach for holistically designing, developing and fielding AI solutions across the engineering lifecycle from data processing to model building, tuning, and training to secure operational deployment
Other
33
stars
8
forks
source link
TASK: Overwrite the ivy user directory in the spark-application chart #400
For java 17 upgrade, we have upgraded the aissemble-spark base image, which has a different home directory than the /opt/spark/ directory so to make sure Ivy cache can find the right directory (Ivy cache is default to the environment's home directory. e.g.: ~/.ivy2), we will need to override the ivy user directory by specifying the spark.jars.ivy directory in the chart.
Currently, to fix the build issue, we have moved the spark.jars.ivy from the spark application chart to the dev values file to avoid the local test issue. However, since our local test is not using the docker container but only copy over the spark configuration from the yaml file to configure the spark session in the local environment, we should not copy spark.jars.ivy directory for the spark tester setup but leave it as default so that it can still use the default home directory and when we deploying the spark application to different environment, we don't need to set spark.jars.ivy path in the values file.
DOD
Acceptance criteria required to complete the work
[ ] Move the spark.jars.ivy configuration to the aissemble-spark-application chart
[ ] In python or java test code, avoid spark.jars.ivy configuration being setup in the spark
Test Strategy/Script
How will this item be verified?
Pull the latest aissemble repo
OTS Only:
Since the aissemble-spark-application-chart is not being pushed up to GitHub packages yet, we will make some local modification
In thefoundation/foundation-mda/src/main/resources/templates/data-delivery-common/pipeline.chart.pom.configuration.vm, modify the line 32 to point to your local directory <<path to your baseline repo>>/extensions/extensions-helm/aissemble-spark-application-chart
In the foundation/foundation-mda/src/main/resources/templates/data-delivery-common/pipeline.chart.pom.configuration.vm, modify the line 24 to be {path = "/path to your baseline repo/extensions/extensions-data-delivery/aissemble-extensions-data-delivery-spark-py", develop = true}
Run mvn clean install -pl :foundation-mda -Dmaven.build.cache.skipCache
Run mvn -B clean install -Parchetype-test -pl :foundation-archetype and verify the build is success.
ewilkins-csi
commented
19 hours ago
Description
For java 17 upgrade, we have upgraded the
aissemble-sparkbase image, which has a different home directory than the/opt/spark/directory so to make sure Ivy cache can find the right directory (Ivy cache is default to the environment's home directory. e.g.: ~/.ivy2), we will need to override the ivy user directory by specifying thespark.jars.ivydirectory in the chart.Currently, to fix the build issue, we have moved the
spark.jars.ivyfrom the spark application chart to the dev values file to avoid the local test issue. However, since our local test is not using the docker container but only copy over the spark configuration from the yaml file to configure the spark session in the local environment, we should not copyspark.jars.ivydirectory for the spark tester setup but leave it as default so that it can still use the default home directory and when we deploying the spark application to different environment, we don't need to setspark.jars.ivypath in the values file.DOD
Acceptance criteria required to complete the work
spark.jars.ivyconfiguration to the aissemble-spark-application chartspark.jars.ivyconfiguration being setup in the sparkTest Strategy/Script
How will this item be verified?
Pull the latest
aissemblerepoOTS Only:
aissemble-spark-application-chartis not being pushed up to GitHub packages yet, we will make some local modificationfoundation/foundation-mda/src/main/resources/templates/data-delivery-common/pipeline.chart.pom.configuration.vm, modify the line 32 to point to your local directory<<path to your baseline repo>>/extensions/extensions-helm/aissemble-spark-application-chartfoundation/foundation-mda/src/main/resources/templates/data-delivery-common/pipeline.chart.pom.configuration.vm, modify the line 24 to be{path = "/path to your baseline repo/extensions/extensions-data-delivery/aissemble-extensions-data-delivery-spark-py", develop = true}mvn clean install -pl :foundation-mda -Dmaven.build.cache.skipCacheRun
mvn -B clean install -Parchetype-test -pl :foundation-archetypeand verify the build is success.References/Additional Context
As needed