search

kubernetes-sigs / kubebuilder

Kubebuilder - SDK for building Kubernetes APIs using CRDs
http://book.kubebuilder.io
Apache License 2.0
7.78k stars 1.44k forks source link

Metadata of PersistentVolumeClaim can not be decoded correctly #2460

Closed hzyfox closed 2 years ago

hzyfox commented 2 years ago

Environment

Kubectl Version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-18T02:59:13Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.4-80+89e0897d2cb807", GitCommit:"89e0897d2cb8073fbb8f700258573f1478d4826a", GitTreeState:"clean", BuildDate:"2021-11-22T03:53:35Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes Version (Kind Cluster)

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.20.7@sha256:cbeaf907fc78ac97ce7b625e4bf0de16e3ea725daf6b04f930bd14c67c671ff9
  - role: worker
    image: kindest/node:v1.20.7@sha256:cbeaf907fc78ac97ce7b625e4bf0de16e3ea725daf6b04f930bd14c67c671ff9
  - role: worker
    image: kindest/node:v1.20.7@sha256:cbeaf907fc78ac97ce7b625e4bf0de16e3ea725daf6b04f930bd14c67c671ff9

Kubebuilder Version

Version: main.version{KubeBuilderVersion:"3.1.0", KubernetesVendor:"1.19.2", GitCommit:"92e0349ca7334a0a8e5e499da4fb077eb524e94a", BuildDate:"2021-05-27T17:54:28Z", GoOs:"darwin", GoArch:"amd64"}

Os

Macos Big Sur 11.6

I use kubebuilder to define my own CRD like below, and it contains VolumeClaimTemplates filed which the type is []coreV1.PersistentVolumeClaim

package v1alpha1
import (
    apps "k8s.io/api/apps/v1"
    coreV1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/util/intstr"
)

type DatabaseSetSpec struct {
        ...
        // +optional
    VolumeClaimTemplates []coreV1.PersistentVolumeClaim `json:"volumeClaimTemplates,omitempty" protobuf:"bytes,4,rep,name=volumeClaimTemplates"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=ami-dbs

// DatabaseSet is the Schema for the databasesets API
type DatabaseSet struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   DatabaseSetSpec   `json:"spec,omitempty"`
    Status DatabaseSetStatus `json:"status,omitempty"`
}

// DatabaseSetStatus defines the observed state of DatabaseSet
type DatabaseSetStatus struct {
    ...
}

//+kubebuilder:object:root=true

// DatabaseSetList contains a list of DatabaseSet
type DatabaseSetList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []DatabaseSet `json:"items"`
}

func init() {
    SchemeBuilder.Register(&DatabaseSet{}, &DatabaseSetList{})
}

But when I apply the CR like the below, I found that the metadata filed is empty

apiVersion: apps.analyticdb.aliyun.com/v1alpha1
kind: DatabaseSet
metadata:
  name: databaseset-sample
spec: 
  ...
  volumeClaimTemplates:
    - metadata:
        name: pvc-test
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: "manual"
        resources:
          requests:
            storage: 3Gi

Here is the yaml which get from the k8s ectd, it cound be found that the metadata of the volumeClaimTemplates is empty.

apiVersion: apps.analyticdb.aliyun.com/v1alpha1
kind: DatabaseSet
metadata:
  creationTimestamp: "2021-12-24T09:46:22Z"
  generation: 1
  name: databaseset-sample
  namespace: default
  resourceVersion: "98727469"
  uid: e64107f2-7a4b-473b-9275-39ab5e2e88dc
spec:
  ...
  volumeClaimTemplates:
  - metadata: {}
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 3Gi
      storageClassName: manual

Does anyone know why？

And when I mark the volumeclaimtemplate field with the below comment

// +kubebuilder:pruning:PreserveUnknownFields
// +kubebuilder:validation:Schemaless

metada can be decoed correctly

Originally posted by @hzyfox in https://github.com/kubernetes-sigs/kubebuilder/discussions/2459

abstractalchemist commented 2 years ago

I believe I'm experiencing a very similar issue and it seems to be connected to how the kubebuilder generates CRDs from the structs defined in the package api/. If I so something like this:

type PodTemplateSpecForExample struct {
  metav1.ObjectMeta `json:"metadata,omitempty"`
  Spec              v1.PodSpec `json:"spec,omitempty"`
}

// ExampleSpec defines the desired state of Example
type ExampleSpec struct {
  // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
  // Important: Run "make" to regenerate code after modifying this file

  // Template For Pod
  Template PodTemplateSpecForExample `json:"template"`
}

In the CRD, this is generated:

          properties:
              template:
                description: Template For Pod
                properties:
                  metadata:
                    type: object
                  spec:
                    description: PodSpec is a description of a pod.
                    properties: ...

But a couple of odd things seem to happen which makes not sense. If I do kubectl explain

ssm-user@ip-172-31-5-82:~/project$ kubectl explain examples.spec.template.metadata
KIND:     Example
VERSION:  examples.my.domain/v1alpha1

DESCRIPTION:
     <empty>
ssm-user@ip-172-31-5-82:~/project$

which was odd. Then kubectl apply on

apiVersion: examples.my.domain/v1alpha1
kind: Example
metadata:
  name: example-sample
spec:
  template:
    metadata:
      labels:
        app: foo
    spec:
      containers:
      - image: centos:8
        name: test

shows up in kubernetes as

apiVersion: examples.my.domain/v1alpha1
kind: Example
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"examples.my.domain/v1alpha1","kind":"Example","metadata":{"annotations":{},"name":"example-sample","namespace":"default"},"spec":{"template":{"metadata":{"labels":{"app":"foo"}},"spec":{"containers":[{"image":"centos:8","name":"test"}]}}}}
  creationTimestamp: "2021-12-27T01:46:58Z"
  generation: 1
  name: example-sample
  namespace: default
  resourceVersion: "88368"
  selfLink: /apis/examples.my.domain/v1alpha1/namespaces/default/examples/example-sample
  uid: 36c38649-f3cd-4e1a-bb20-b7e8b2d773c6
spec:
  template:
    metadata: {}
    spec:
      containers:
      - image: centos:8
        name: test

As you can see, the metadata is completely missing. It seems as if the generation is deliberately missing or ignoring any other declaration of ObjectMeta in the type if it doesn't appear in the top-level struct.

hzyfox commented 2 years ago

@abstractalchemist I have found the solution, use the controller-gen crd option crd:generateEmbeddedObjectMeta=true will work

abstractalchemist commented 2 years ago

That's moderately better, but I think this needs to be better documented as it's not explained anywhere ( I don't even see this option in the kubebuilder book ) and it's not clear to me why this would be specifically called out as ignored for processing by the builder.

hzyfox commented 2 years ago

@abstractalchemist I found this option through controller-gen -h, and there is no mention of this option in the official kubebuilder controller-gen CLI documention .

hzyfox commented 2 years ago

@abstractalchemist And I think crd:generateEmbeddedObjectMeta shoul default to be true, I don’t know why this option is turned off by default. CRD is very likely to use nested ObjectMetadata

camilamacedo86 commented 2 years ago

HI @hzyfox and @abstractalchemist,

We would like to have a FAQ section on the docs : https://github.com/kubernetes-sigs/kubebuilder/issues/1723

WDYT about to collab with the project by creating this one and adding it there? The idea would be similar to: https://sdk.operatorframework.io/docs/faqs/

hzyfox commented 2 years ago

@camilamacedo86 LGTM, it will be useful to add it to FAQ. But I don’t know how to describe this problem accurately and concisely

camilamacedo86 commented 2 years ago

@hzyfox, could you try to contact the controller-tools maintainers and ask a help over how better you can describe this scenario? Maybe @alvaroaleman can give a hand for us here.

TIEDPAG commented 2 years ago

I have the same problem but I don't see crd:generateEmbeddedObjectMeta option

controller-gen version

Version: v0.4.1

controller-gen help

+crd[:allowDangerousTypes=<bool>][,crdVersions=<[]string>][,maxDescLen=<int>][,preserveUnknownFields=<bool>][,trivialVersions=<bool>]  package  generates CustomResourceDefinition objects.
TIEDPAG commented 2 years ago

Tried it, this parameter is available in controller-gen 0.6.2

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/kubebuilder/issues/2460#issuecomment-1177359137): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.