apache-spark-on-k8s / userdocs

Repo to host documentation site for Apache Spark on Kubernetes
Apache License 2.0
10 stars 6 forks source link

Correct dynamic shuffling example #19

Closed johscheuer closed 7 years ago

johscheuer commented 7 years ago

The docs for dynamic shuffling are not working the referenced docker image doesn't exist:

docker pull kubespark/spark-driver:latest
Error response from daemon: manifest for kubespark/spark-driver:latest not found

This results in the following error:

    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                  From                                                   Message
  ----     ------                 ----                 ----                                                   -------
  Normal   Scheduled              36m                  default-scheduler                                      Successfully assigned group-by-test-1508830761433-driver to gke-big-data-demo-default-pool-cbb0305a-skzj
  Normal   SuccessfulMountVolume  36m                  kubelet, gke-big-data-demo-default-pool-cbb0305a-skzj  MountVolume.SetUp succeeded for volume "default-token-fs5zk"
  Normal   Pulling                4m (x11 over 36m)    kubelet, gke-big-data-demo-default-pool-cbb0305a-skzj  pulling image "kubespark/spark-driver:latest"
  Warning  Failed                 4m (x11 over 36m)    kubelet, gke-big-data-demo-default-pool-cbb0305a-skzj  Failed to pull image "kubespark/spark-driver:latest": rpc error: code = 2 desc = Error response from daemon: {"message":"manifest for kubespark/spark-driver:latest not found"}
  Warning  FailedSync             16s (x161 over 36m)  kubelet, gke-big-data-demo-default-pool-cbb0305a-skzj  Error syncing pod
  Normal   BackOff                16s (x150 over 36m)  kubelet, gke-big-data-demo-default-pool-cbb0305a-skzj  Back-off pulling image "kubespark/spark-driver:latest"

If you adjust the docker image tag everything works as expected:

$ kubectl get po --show-all          
NAME                                 READY     STATUS      RESTARTS   AGE
group-by-test-1508833005822-driver   0/1       Completed   0          56s
shuffle-cjcsw                        1/1       Running     0          39m
shuffle-hhqj3                        1/1       Running     0          39m
shuffle-kl4d0                        1/1       Running     0          39m
shuffle-vkcxz                        1/1       Running     0          39m
mccheah commented 7 years ago

We should probably be publishing a latest tag anyways though - @ash211 @foxish for thoughts.

foxish commented 7 years ago

@johscheuer, the images and tags are documented in https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html. I'd rather we encourage the use of versioned tags - deployments inexplicably break if they rely on tags like latest, instead of immutable versioned ones.

foxish commented 7 years ago

This change LGTM. We ought to bring the docs here in sync with that in the docs repo.

foxish commented 7 years ago

oops.. I just realized that this is the docs repo. My bad. Looks like we just missed updating it - merging; whether we should be pushing a separate latest tag is a separate discussion.