spark on k8s

[GunSik2]

Spark 예제 앱 이미지 생성

• download spark

  wget https://mirror.navercorp.com/apache/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
  tar xvzf spark-3.1.2-bin-hadoop3.2.tgz

• set env

  cd spark-3.1.2-bin-hadoop3.2
  export SPARK_HOME=$PWD
  alias spark-shell=”$SPARK_HOME/bin/spark-shell”

• create spark app docker image

  $SPARK_HOME/bin/docker-image-tool.sh \
  -r cgshome2 -t v3.0.1-j14 \
  -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile  \
  -b java_image_tag=14-slim build

• check image

  $ docker images

• push image

  docker login
  docker push cgshome2/spark-py:v3.0.1-j14
  docker push cgshome2/spark:v3.0.1-j14

  Kubernetes 배포

• k8s info

  $ kubectl cluster-info
  Kubernetes control plane is running at https://127.0.0.1:6443
  CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/rke2-coredns-rke2-coredns:udp-53/proxy

• k8s 배포

  
  // snap install openjdk (JDK 설치 후)
  // export JAVA_HOME=/snap/openjdk/current/jdk/
  // export PATH=$PATH:$JAVA_HOME/bin/

$SPARK_HOME/bin/spark-submit \ --master k8s://https://127.0.0.1:6443 \ --deploy-mode cluster \ --name spark-pi \ --class org.apache.spark.examples.SparkPi \ --conf spark.executor.instances=5 \ --conf spark.kubernetes.container.image=cgshome2/spark:v3.0.1-j14 \ local:///home/steward/Downloads/spark-3.1.2-bin-hadoop3.2/examples/jars/spark-examples_2.12-3.1.2.jar

참고
- https://ichi.pro/ko/kubernetes-ui-spark-67629249853531
- https://spark.apache.org/docs/latest/

[GunSik2]

• spark-submit 실행 오류

  Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make private java.nio.DirectByteBuffer(long,int) accessible: module java.base does not "opens java.nio" to unnamed module @654001d8
  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
  at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
  at java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:188)
  at java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:181)
  at org.apache.spark.unsafe.Platform.<clinit>(Platform.java:56)
  ... 13 more

• jdk version 8/11 에 한해 지원
• https://stackoverflow.com/questions/67332626/spark-shell-command-giving-errorwindows-cmd-and-cygwin

[GunSik2]

• spark-submit 실행 오류

  
  21/09/14 19:57:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  21/09/14 19:57:33 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
  Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: JcaPEMKeyConverter is provided by BouncyCastle, an optional dependency. To use support for EC Keys you must explicitly add this dependency to classpath.
  at io.fabric8.kubernetes.client.internal.CertUtils.handleECKey(CertUtils.java:163)

[GunSik2]

7 활용하여 배포