search

stackabletech / spark-k8s-operator

Operator for Apache Spark-on-Kubernetes for Stackable Data Platform
https://stackable.tech
Other
51 stars 3 forks source link

Investigate and potentially add support for spark connect #284

Open razvan opened 1 year ago

razvan commented 1 year ago

Spark Connect

Spark 3.5 introduces a new client called Spark Connect.

The use case seems to be thin clients that connect to a running spark driver.

This probably means that the operator needs to be able to start spark connect servers without spark applications and publish a service for "connect" clients.