Setting up Kubernetes

[amineKammah]

Done:

• Configure and install Kubernetes on an EC2 instance
• Create a K8s master and 1 node setup
• Submit a sample job
• Create YAML file to submit the spark job
• Run ocr_service.py on a pod, submitted using a kubernetes job yaml file. (kubectl apply -f ocr_service.yaml) Pods cannot share multiple nodes, therefore this job can only run on one worker node. Therefore I have to submit it using spark-submit to make it run on multiple nodes.

TO Dos:

• Setting up monitoring and scalability

[amineKammah]

Run 2 EC2 machines, Allow all TCP traffic for now.

Setup: sudo su - ufw disable swapoff -a; sed -i '/swap/d' /etc/fstab cat >>/etc/sysctl.d/kubernetes.conf<<EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sysctl --system apt-get update apt install default-jdk scala git -y wget https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz tar xvf spark-3.0.1-bin-hadoop2.7.tgz mv spark-3.0.1-bin-hadoop2.7 /opt/spark echo "export SPARK_HOME=/opt/spark" >> ~/.profile echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.profile echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile source ~/.profile apt-get install docker.io systemctl enable docker systemctl status docker curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main" apt-get install kubeadm kubelet kubectl apt-mark hold kubeadm kubelet kubectl apt install python3-pip apt install python-is-python3 pip3 install pytesseract Pillow

Master setup: kubeadm init kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f https://docs.projectcalico.org/v3.14/manifests/calico.yaml kubeadm token create --print-join-command

[amineKammah]

Submit spark job to master node in client mode

./bin/spark-submit \ --master k8s://https://172.31.45.129:6443\ --deploy-mode cluster \ --name spark-py \ --conf spark.executor.instances=1 \ --conf spark.kubernetes.container.image=kammahm/spark-py \ local:///ensimag-sdtd/data_processing/ocr_service.py

[amineKammah]

Setting up tutorial: https://itnext.io/kubernetes-part-2-a-cluster-set-up-on-aws-with-aws-cloud-provider-and-aws-loadbalancer-f02c3509f2c2