Note that 1) Spark 2.4 will not work with Python 3.8 and so needs to be upgraded first to 3.x, and 2) in the loader container openjdk-8-jre-headless is not available for a 3.8-buster base image so will need to happen once jdk version 11 can be used. This assumes that having different versions of Java in the loader and spark containers would be problematic, but perhaps they can be done independent of each other.
dolsysmith
commented
3 years ago
Note that 1) Spark 2.4 will not work with Python 3.8 and so needs to be upgraded first to 3.x, and 2) in the loader container openjdk-8-jre-headless is not available for a 3.8-buster base image so will need to happen once jdk version 11 can be used. This assumes that having different versions of Java in the loader and spark containers would be problematic, but perhaps they can be done independent of each other.