-
k8s: GKE 1.17 with auto-scaling node-pool with taint
spark-operator: v1beta2-1.2.0-3.0.0
SparkApp:
dynamicAllocation: enabled
tolerations and nodeSelector: specified
Sometimes executors get…
-
### What happened?
I am trying to deploy a Beam job (Python Beam) that runs on a PortableRunner (Flink Runner) in my Kubernetes cluster.
I have not experienced issues prior with Beam using the Flin…
-
Hi,
I'm adding the operator and sparkapplication helm charts to my auto deploy script, among other components (e.g. tomcat, zookeeper, etc). Since sparkapplication crds are defined by the operator,…
-
### Description
We are replacing presto in the system with velox, but we have done a lot of self-research and optimization on performance based on presto. We found that there is a performance gap b…
-
### Issues Policy acknowledgement
- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)
### Willi…
-
Trying to run a spark operator, i am using the `pi.py` file and `spark-py-pi.yaml` files
```
import sys
from random import random
from operator import add
from pyspark.sql import SparkSession…
-
If we use the `prometheus operator`, we can easily configure the target pod we want to collect metrics for using `pod monitor` crd and label selector and deploy the `prometheus server`.
The approxima…
-
Hi,
we have 10 scheduled spark operators running every 10 minutes in the same namespace. Sometimes one of them gets stuck, we can see that operator is cycling through executors as they crash on st…
-
In order to deploy PVC with every spark executor, it is needed to use multiple configurations mentioned in spark documentation -
From spark documentation, these are the needed configurations to use…
-
ETA: 2024-06-30
We want to use IPv4 addresses of SPARK nodes as the scarce resource that makes it expensive for a single party to run many nodes. ATM, we rely on the trusted spark-api service to re…