search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

SparkJob resource in Kubernetes #111

Open foxish opened 7 years ago

foxish commented 7 years ago

We are planning to make changes to the way external resources are represented in Kubernetes. This includes TPRs which we planned on using for representing a SparkJob.

I'm using this issue as a place to collect thoughts about what items we want a spark job to surface up to Kubernetes.

A preliminary list includes the following:

Do we have other items we want Kubernetes to know about?

cc/ @erikerlandson @mccheah @ash211 @ssuchter

foxish commented 7 years ago

This is prerequisite to https://github.com/apache-spark-on-k8s/spark/issues/22

ash211 commented 7 years ago

URLs of driver and executor UIs are most valuable to me.

An additional point to note is that when a job is running the Spark UI is served out of those services, but after those services stop they also stop serving the Spark UI. Often you want to look at a Spark UI after its job has finished. So Spark supports writing event logs out to a separate location (spark.eventLog.dir) and running a separate service called the Spark History Server that recreates most of the Spark UI for offline use. This is documented at http://spark.apache.org/docs/latest/monitoring.html#viewing-after-the-fact

For SparkJobs after they've stopped, I'd want to also have the URL of the job in the Spark History Server as a link.

kimoonkim commented 7 years ago

+1 for Spark web UI URL.

As to spark.eventLog.dir, this is typically a dir on HDFS, as the example on the documentation page shows. So this might not work right away:

spark.eventLog.dir hdfs://namenode/shared/spark-logs

ssuchter commented 7 years ago

Number of executors active - that's interesting because it might mutate frequently - is that something that can be supported by this Kubernetes external resource (frequently mutating items)

If so, it'd be pretty snazzy to also include number of tasks active and what stage number (min/max) are running right now. (I suggest min & max because the full list might be an unbounded set, and that way you can just show two scalars).

URLs are really important, as Andrew pointed out.

Interesting thing for this whole project is to make the Spark History server work. Do we need an external HDFS for that? Or will some other external storage work?

The other thing that occurs to me a-la the first bullet is the main Spark class name that we're trying to run. (What the main class name in the user's code) For example 'org.apache.spark.examples.SparkPi'.

I also bet that the user will want to know what Kubernetes namespace this was run in, but that seems like it might be known elsewhere from this external resource spec.

foxish commented 7 years ago

Given that we have some time now, we should add TPRs now rather than later. I'm marking this as alpha. The lack of the TPR installed will not block a user from submitting jobs, but if the endpoint is present, we will create such an object and possibly update it.

A slew of fixes for kubectl to work with TPR are coming in k8s 1.6, and to have it around from the beginning would be a good idea. I'll incorporate the above items and we can do some rounds of review.

Most of the effort is just porting it over from https://github.com/foxish/spark

cc/ @iyanuobidele

iyanuobidele commented 7 years ago

@foxish

+1 I am on board with this. I believe you discussed this with the team at the latter part of the sig meeting.

I like the idea - Whenever the upstream fixes come, the capability will be there for cluster admins to leverage.

I'll take this on, since we might need to make a few changes to the SparkJob abstraction.

foxish commented 7 years ago

Sounds good. Thanks @iyanuobidele!

foxish commented 7 years ago

Reviving this issue. CRDs are now ready for consumption in Kubernetes 1.8. I'll be reworking @iyanuobidele's old PR and sending one using CRDs early next week. It'll be flag gated for now because using it in Kubernetes 1.7 and below has some issues, but should be fine 1.8+, but I'd like to get it upstreamed (but after the critical pieces are) because it adds some missing UX pieces.

cc @apache-spark-on-k8s/contributors