A Spark Kubernetes deployment

[Jean-Baptiste-Lasselle]

Here are some references to deploy a spark stack to kubernetes:

• with quick research on github, I could witness that even today, a production ready spark stack does not exist, in open source. see https://github.com/search?q=spark+kubernetes&type=repositories&p=2
• I read a few articles which confirmed my first perception of that state of the opensource market: In simple words, you today have only two serious options to deploy spark to kubernetes, either the spark official integration, or Google's kubeflow spark operator. It seems like I will have to experiment both.
• Articles / resources I found on the deploying with either of those two options:
  • the spark source code: https://github.com/apache/spark
  • a short simple post about it: https://medium.com/@SaphE/deploying-apache-spark-on-a-local-kubernetes-cluster-a-comprehensive-guide-d4a59c6b1204
  • a 3 episodes articles "My journey deploying spark to kubernetes", interesting: https://dev.to/stack-labs/my-journey-with-spark-on-kubernetes-in-python-1-3-4nl3
  • the spark documentation page about kubernetes deployment (surprisingly poor): https://spark.apache.org/docs/latest/running-on-kubernetes.html
  • Google's kubeflow Spark Operator (which will apparently always stay in beta version): https://github.com/kubeflow/spark-operator

The first question i have in mind after this quick investigation is : Ok spark works on a filesystem, and i don't see anywhere how this filesystem is provisioned ?

It seems that hdfs is related to S3: