-
Wrote up my thoughts so far. I've pivoted away from the separate "external shuffle service manager" component because it seems like we can accomplish most of what we want without it. Comments and edit…
-
-
Hello,
as we work with my team with the spark UI, we are facing the following problem:
- we use the ability of the spark operator to create an ingress
- we enable ssl for spark UI with sparkConf …
-
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.8.3
### What happened?
I'm having an issue when using SparkKubern…
-
Once https://github.com/apache/spark/pull/21669/files is pulled into the mainline spark, there will be kerberos support for HDFS clusters that use kerberos for authentication.
I wasn't able to find…
-
Hi,
i had a look at https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-2/ but something is not clear:
according to https://databricks.com/session_na20/native-support-of-prometheus-monitori…
-
I updated my delta lake to version 3.0.0 and user pyspark 3.5.0.
I Use this code to save my df in delta lake:
```
df\
.write\
.format("delta")\
.saveAsTable(
…
-
## Description
We use the operator to manage the lifecycle of both batch and Spark streaming applications. Streaming apps in particular are long lived and if using dynamic allocation may scale up a…
-
I want to run Spark on Kubernetes with Dynamic Resource Allocation, so I have to start a shuffle service. However, I don't find any example on how to do this. Can anyone post a sample yaml file to sho…
-
Hi thanks for the work, spark-on-k8s-operator really makes thing easier.
It seems the operator currently does not support specifying a custom serviceAccount for executors. I tried both serviceAccou…