search

JahstreetOrg / spark-on-kubernetes-helm

Spark on Kubernetes infrastructure Helm charts repo
Apache License 2.0
199 stars 76 forks source link

Fix Livy StatefulSet api version `apps/v1beta2 -> apps/v1` #31

Closed liubarnabas closed 4 years ago

jahstreet commented 4 years ago

Hi @liubarnabas , thank you for the contribution. Is there any specific reason for this change?

liubarnabas commented 4 years ago

Need upgrade from "apps/v1beta2" to "apps/v1", otherwise error will show up, during the deploy.

jahstreet commented 4 years ago

@liubarnabas , which K8s API version your cluster uses?

liubarnabas commented 4 years ago

I use latest EKS in AWS. kubectl api-versions admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 batch/v1 batch/v1beta1 certificates.k8s.io/v1beta1 coordination.k8s.io/v1 coordination.k8s.io/v1beta1 crd.k8s.amazonaws.com/v1alpha1 events.k8s.io/v1beta1 extensions/v1beta1 metrics.k8s.io/v1beta1 monitoring.coreos.com/v1 networking.k8s.io/v1 networking.k8s.io/v1beta1 node.k8s.io/v1beta1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 scheduling.k8s.io/v1 scheduling.k8s.io/v1beta1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1

jahstreet commented 4 years ago

Then it should be 1.16.8. Note that this chart (if you are going to use the default Docker images) is guaranteed to work properly with K8s API <= 1.15.3 due to the Spark v2.4.5 fabric8 Kubernetes client version limitations. This will change with the later Spark versions. Rather than that I'll test it with the previous K8s versions and merge if it is ok.

jahstreet commented 4 years ago

Hi @liubarnabas , https://github.com/jahstreet/spark-on-kubernetes-helm/pull/31/commits/a93fe166023f47e2aef862b3cfeb570f5a6b9b33 is unrelated to this PR. Would you mind to merge it separately? EDIT" oh, I see the separate PR https://github.com/jahstreet/spark-on-kubernetes-helm/pull/32, will do the houskeeping on my own, no actions required from your side. Thx for the contribution.

pdambrauskas commented 4 years ago

Hey :), are there any plans to release this change? According to K8s documentation apps/v1 should be available since v1.9.

StatefulSet in the apps/v1beta1 and apps/v1beta2 API versions is no longer served Migrate to use the apps/v1 API version, available since v1.9. Existing persisted data can be retrieved/updated via the new version.

https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/