Measuring performance - Githubissues

amineKammah / ensimag-sdtd

0 stars 0 forks source link

Measuring performance #4

Open amineKammah opened 4 years ago

amineKammah commented 4 years ago

Done:

Creating a Spark scalable distributed cluster.
Creating a simple demo app allowing to measure performance.

TODO:

Add 1, 2, 3 workers nodes and compare performance and measure scalability

amineKammah commented 4 years ago

Submitting job using:

./bin/spark-submit
--master k8s://https://172.31.45.129:6443\
--deploy-mode cluster
--name spark-pi
--conf spark.executor.instances=1
--conf spark.kubernetes.container.image=kammahm/spark-py
local:///ensimag-sdtd/data_processing/ocr_service.py

if getting error message: insufficient memory and cpu, restart master-node