Spark wasn't getting rolled during "gen3 roll all". We decided it would be best for Spark to be a sidecar image instead of running it in a separate pod. This way spark will only run if an ETL job is triggered and it will always pull the image defined in the manifest.json file.
Improvements
Spark wasn't getting rolled during "gen3 roll all". We decided it would be best for Spark to be a sidecar image instead of running it in a separate pod. This way spark will only run if an ETL job is triggered and it will always pull the image defined in the manifest.json file.