kubeflow spark operator & SparkHistoryService on k8s - spark driver/executor logs not showing up Spark History Server

[karanalang]

• [ ] ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

I have kubeflow spark operator installed on GKE (in namespace - so350), as well as Spark History Server installed on GKE in namespace shs-350. The spark job is launched in a separate names - spark-apps.

When i launch the spark job, it runs fine and i'm able to see the job details in the Spark History server UI. The Spark History Server is configured to have the events stored in GCP storage bucket.

While the events are getting stored, the worker & executor logs are not getting stored in the strage bucket and hence not showing up in the History Server UI. i.e. the stderr/etdout links are not enabled on the Spark History Server UI.

How do i enable storing the worker/executor logs on GCP bucket ? Do i need to install fluentbit ot fluentd to collect the logs from k8s pods & store in the storage bucket ?

Any inputs on this ?

tia!

Pls note : to enable the spark events to flow into GCP storage bucket, CM is created as shown below to update the spark-defaults.conf

apiVersion: v1
kind: ConfigMap
metadata:
  name: spark-history-server-conf
  namespace: shs-350
data:
  spark-defaults.conf: |
    spark.history.fs.logDirectory=gs://<storage-bucket>/spark-events
    spark.hadoop.google.cloud.auth.service.account.enable=true
    spark.hadoop.google.cloud.auth.service.account.json.keyfile=/etc/secrets/spark-gcs-key.json

In the spark job yaml, following configuration is done :

"spark.eventLog.enabled": "true"
"spark.eventLog.dir": "gs://<storage-bucket>/spark-events"

[jacobsalway]

Hey, I don't think logs are shipped as part of the event logs that the driver writes to the event storage (GCS in your case). I haven't gone through the source code in Spark core but the Spark docs don't recommend any history server specific solution (https://spark.apache.org/docs/latest/running-on-kubernetes.html#accessing-logs).

Logs are probably handled differently in other ways of running Spark e.g. standalone mode or via Yarn. I'd recommend using a more generic log collection solution that just tails the container stdout/stderr e.g. something managed like Datadog, or an open source stack that you run yourself.

[vikas-saxena02]

The default behaviour in spark on kubernetes is to redirect the spark logs to the pod logs so you can only access them through kubectl or sparkctl.