-
I am not too sure if this is an issue with Spark/K8S or the entry-point of the image, but the result is that the driver can not access the files passed through `--py-files` and `--files` when these ar…
-
### Describe the bug
We easily run into this problem when running queries with `spark.comet.exec.shuffle.mode=jvm`:
```
org.apache.spark.memory.SparkOutOfMemoryError: Unable to acquire 67108848…
-
Platform: `minikube version: v1.24.0`
tl;dr airflow won't start, logs of everything are listed below
I'm trying to recreate everything and I'm stuck with this part. I've been waiting for some t…
-
I have a problem changing the flink application in my pipeline. The problem is this: for example, I want to change the amount of memory for the task мanager in one of the job clusters. I change the d…
-
- [ ] ✋ I have searched the open/closed issues and my issue is not listed.
I am trying to deploy a Spark application on Kubernetes using the Spark Operator, but I'm encountering an issue related to…
-
### What happened?
I am trying to deploy a Beam job (Python Beam) that runs on a PortableRunner (Flink Runner) in my Kubernetes cluster.
I have not experienced issues prior with Beam using the Flin…
-
See discussion in #719 with this last open issue. Moving to a separate issue for clarity.
> > How can you launch a spark application and then reliably wait for it to finish?
>
> I would explore…
-
Example: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml
Operator values
```
operatorVersion: v1beta2-1.2.1-3.0.0
imagePullSecrets:
- name: ha…
-
ETA: 2024-06-30
We want to use IPv4 addresses of SPARK nodes as the scarce resource that makes it expensive for a single party to run many nodes. ATM, we rely on the trusted spark-api service to re…
-
cc @Fokko .
This is a super simple implementation of an iceberg client for dask. It works for the limited couple of datasets I have available including
- version metadata choice
- snapshot time …