stackabletech / spark-k8s-operator

Operator for Apache Spark-on-Kubernetes for Stackable Data Platform
https://stackable.tech
Other
51 stars 2 forks source link

Add support for deploying stand alone Spark clusters #203

Open razvan opened 1 year ago

razvan commented 1 year ago

Description

Jupyter notebooks can connect to stand alone clusters. This allows for much better scalability when analyzing big data with Pyspark notebooks.

Acceptance

sbernauer commented 1 year ago

I personally like Notebooks running in client mode and requesting executors and would prefer that solution over a Spark standalone cluster. We used that extensively at my previous company