AdvancedStatefulSet CRD won't load in GKE v1.16.9 :: spec.preserveUnknownFields: Invalid value: true

[donbowman]

Bug Report

What version of Kubernetes are you using?

kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-gke.2", GitCommit:"4751ff766b3f6dbf6c6a63394a909e8108e89744", GitTreeState:"clean", BuildDate:"2020-05-08T16:44:50Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

v1.1.2

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

$ kubectl get sc NAME PROVISIONER AGE managed-premium kubernetes.io/gce-pd 339d standard (default) kubernetes.io/gce-pd 339d

What's the status of the TiDB cluster pods?

What did you do?

From instructions on https://docs.pingcap.com/tidb-in-kubernetes/v1.1/advanced-statefulset page,

kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/advanced-statefulset-crd.v1.yaml --validate=false
The CustomResourceDefinition "statefulsets.apps.pingcap.com" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

What did you expect to see?

expected CRD to load

What did you see instead? The CustomResourceDefinition "statefulsets.apps.pingcap.com" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

[cofyc]

it seems x-kubernetes-preserve-unknown-fields can't be used if the defaulting is enabled. I'll take a look.

as a workaround solution, you can use advanced-statefulset-crd.v1beta1.yaml in Kubernetes 1.16.

[cofyc]

I can't reproduce this issue with GKE 1.16, here is output on my newly created cluster:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a17149e1a189050796ced469dbd78d380f2ed5ef", GitTreeState:"clean", BuildDate:"2020-04-16T11:44:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-gke.6", GitCommit:"14bc8ad5f8c245f1240a8e4eab128c4d51bfeffe", GitTreeState:"clean", BuildDate:"2020-05-30T02:07:03Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
$ kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/advanced-statefulset-crd.v1.yaml
customresourcedefinition.apiextensions.k8s.io/statefulsets.apps.pingcap.com created

Can you run the following commands again and paste the output?

kubectl version
kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/master/manifests/advanced-statefulset-crd.v1.yaml

[donbowman]

now that the object exists, it was able to overwrite it w/o trouble. let me try on another cluster that has not had it installed.

this cluster was upgraded from 1.15 to 1.16 (and had the v1beta1 version of this crd installed when it was at 1.15)

[donbowman]

r$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-17T11:41:22Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-gke.2", GitCommit:"4751ff766b3f6dbf6c6a63394a909e8108e89744", GitTreeState:"clean", BuildDate:"2020-05-08T16:44:50Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}
don@cube[ca-1]:tidb-operator$ kubectl apply -f advanced-statefulset-crd.v1.yaml 
The CustomResourceDefinition "statefulsets.apps.pingcap.com" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema

it happens when the v1beta1 is present, and a crd is using the v1beta1. e.g. when it wants to upgrade.

[donbowman]

$ kubectl -n titan get statefulsets.apps.pingcap.com
NAME      DESIRED   CURRENT   AGE
db-pd     4         3         88d
db-tidb   0                   88d
db-tikv   4         3         88d

yes, the problem is is you have existing v1beta1 objects, you cannot install the new crd since it has the problem w/ preserveUnknown.

[cofyc]

Is this a Kubernetes bug? I created an issue in advanced-statefulset repo. I'm not sure what's the right way to upgrade the CRD in this case...

[donbowman]

https://kubernetes.io/blog/2019/06/20/crd-structural-schema/

i think you are missing the schema, are relying on the old schema-less model.

Any new feature for CRDs starting from Kubernetes 1.15 will require to have a structural schema:

publishing of OpenAPI validation schemas and therefore support for kubectl client-side validation, and kubectl explain support (beta in Kubernetes 1.15)
CRD conversion (beta in Kubernetes 1.15)
CRD defaulting (alpha in Kubernetes 1.15)
Server-side apply (alpha in Kubernetes 1.15, CRD support pending).
Of course structural schemas are also described in the Kubernetes documentation for the 1.15 release.

https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema

[cofyc]

Thanks!