Convert to sidecar

[agathanatasha]

To Deploy

juju deploy ./spark.charm --resource oci-image=ghcr.io/googlecloudplatform/spark-operator:v1beta2-1.3.7-3.1.1 --trust

To create example sparkapp

kubectl create -f ./example/spark-pi.yaml

If you want to create other spark apps, make sure to update .spec.driver.serviceAccout to -driver-account, i.e. spark-k8s-driver-account

changes made

• updated crds
• added rbac.yaml manifests as it requires a separate serviceaccount for the sparkapp driver
• updated example sparkapp with correct serviceaccount, fix #17
• updated tox
• updated ci to use 1.22/stable
• added unit tests
• remove all k8s resources created, fix #20

changes not included

• prometheus relations. The current relation doesn't use the charm lib. Would update that in a separate PR.

[github-actions[bot]]

Libraries are not up to date with their remote counterparts. If this was not intentional, run charmcraft fetch-libs and commit the updated libs to your PR branch.

stdout

``` Library charms.observability_libs.v1.kubernetes_service_patch updated to version 1.3. ```

stderr

``` ```

[github-actions[bot]]

Libraries are not up to date with their remote counterparts. If this was not intentional, run charmcraft fetch-libs and commit the updated libs to your PR branch.

stdout

``` Library charms.observability_libs.v1.kubernetes_service_patch updated to version 1.3. ```

stderr

``` ```

[github-actions[bot]]

Libraries are not up to date with their remote counterparts. If this was not intentional, run charmcraft fetch-libs and commit the updated libs to your PR branch.

stdout

``` Library charms.observability_libs.v1.kubernetes_service_patch updated to version 1.3. ```

stderr

``` ```

[github-actions[bot]]

Libraries are not up to date with their remote counterparts. If this was not intentional, run charmcraft fetch-libs and commit the updated libs to your PR branch.

stdout

``` Library charms.observability_libs.v1.kubernetes_service_patch updated to version 1.3. ```

stderr

``` ```

[ca-scribner]

Not sure if this is a spark configuration thing or just something that'll happen with the example workload, but if I deploy spark and then execute the example workflow I get a SparkApplication that says Completed but also has a warning event saying it was missing a container (kubectl describe SparkApplication). I'm not sure if that is just a transient thing or something bigger?

[agathanatasha]

Not sure if this is a spark configuration thing or just something that'll happen with the example workload, but if I deploy spark and then execute the example workflow I get a SparkApplication that says Completed but also has a warning event saying it was missing a container (kubectl describe SparkApplication). I'm not sure if that is just a transient thing or something bigger?

These are the events I see. There are no warnings. Could you paste what you see?

Events:
  Type    Reason                     Age              From            Message
  ----    ------                     ----             ----            -------
  Normal  SparkApplicationAdded      24s              spark-operator  SparkApplication spark-pi was added, enqueuing it for submission
  Normal  SparkApplicationSubmitted  21s              spark-operator  SparkApplication spark-pi was submitted successfully
  Normal  SparkDriverRunning         18s              spark-operator  Driver spark-pi-driver is running
  Normal  SparkExecutorPending       15s              spark-operator  Executor [spark-pi-3f1643825f91b4fe-exec-1] is pending
  Normal  SparkExecutorRunning       13s              spark-operator  Executor [spark-pi-3f1643825f91b4fe-exec-1] is running
  Normal  SparkDriverCompleted       9s (x2 over 9s)  spark-operator  Driver spark-pi-driver completed
  Normal  SparkApplicationCompleted  9s               spark-operator  SparkApplication spark-pi completed
  Normal  SparkExecutorCompleted     8s (x2 over 8s)  spark-operator  Executor [spark-pi-3f1643825f91b4fe-exec-1] completed