kubernetes / kubernetes

Production-Grade Container Scheduling and Management
https://kubernetes.io
Apache License 2.0
110.16k stars 39.43k forks source link

Custom Resource Object field, of type object (map), not reflecting in the retrieved resource #98683

Closed nitishm closed 3 years ago

nitishm commented 3 years ago

What happened: Custom resource type has an embedded field Values of golang type map[string]interface{} or object in OAS3. On applying an object with the field set to a valid YAML block, the objects gets submitted successfully but on retrieving the object back, the values field is set to an empty object.

What you expected to happen: Fields should be reflected/set in the applied object and not be defaulted to an empty {} object

How to reproduce it (as minimally and precisely as possible):

file: application-crd.yaml

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: (unknown)
  creationTimestamp: null
  name: applications.orkestra.azure.microsoft.com
spec:
  group: orkestra.azure.microsoft.com
  names:
    kind: Application
    listKind: ApplicationList
    plural: applications
    singular: application
  scope: Cluster
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Application is the Schema for the applications API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ApplicationSpec defines the desired state of Application
            properties:
              chart:
                properties:
                  chartPullSecret:
                    description: ChartPullSecret holds the reference to the authentication
                      secret for accessing the Helm repository using HTTPS basic auth.
                      NOT IMPLEMENTED!
                    properties:
                      name:
                        type: string
                    required:
                    - name
                    type: object
                  git:
                    description: Git URL is the URL of the Git repository, e.g. `git@github.com:org/repo`,
                      `http://github.com/org/repo`, or `ssh://git@example.com:2222/org/repo.git`.
                    type: string
                  name:
                    description: Name is the name of the Helm chart _without_ an alias,
                      e.g. redis (for `helm upgrade [flags] stable/redis`).
                    type: string
                  path:
                    description: Path is the path to the chart relative to the repository
                      root.
                    type: string
                  ref:
                    description: Ref is the Git branch (or other reference) to use.
                      Defaults to 'master', or the configured default Git ref.
                    type: string
                  repository:
                    description: RepoURL is the URL of the Helm repository, e.g. `https://kubernetes-charts.storage.googleapis.com`
                      or `https://charts.example.com`.
                    type: string
                  secretRef:
                    description: SecretRef holds the authentication secret for accessing
                      the Git repository (over HTTPS). The credentials will be added
                      to an HTTPS GitURL before the mirror is started.
                    properties:
                      name:
                        type: string
                      namespace:
                        type: string
                    required:
                    - name
                    type: object
                  skipDepUpdate:
                    description: SkipDepUpdate will tell the operator to skip running
                      'helm dep update' before installing or upgrading the chart,
                      the chart dependencies _must_ be present for this to succeed.
                    type: boolean
                  version:
                    description: Version is the targeted Helm chart version, e.g.
                      7.0.1.
                    type: string
                type: object
              disableOpenAPIValidation:
                description: DisableOpenAPIValidation controls whether OpenAPI validation
                  is enforced.
                type: boolean
              forceUpgrade:
                description: Force will mark this Helm release to `--force` upgrades.
                  This forces the resource updates through delete/recreate if needed.
                type: boolean
              groupID:
                type: string
              helmVersion:
                description: 'HelmVersion is the version of Helm to target. If not
                  supplied, the lowest _enabled Helm version_ will be targeted. Valid
                  HelmVersion values are: "v2", "v3"'
                enum:
                - v2
                - v3
                type: string
              maxHistory:
                description: MaxHistory is the maximum amount of revisions to keep
                  for the Helm release. If not supplied, it defaults to 10.
                type: integer
              namespace:
                description: Namespace to which the HelmRelease object will be deployed
                type: string
              releaseName:
                description: ReleaseName is the name of the The Helm release. If not
                  supplied, it will be generated by affixing the namespace to the
                  resource name.
                type: string
              repo:
                description: ChartRepoNickname is used to lookup the repository config
                  in the registeries config map
                type: string
              resetValues:
                description: ResetValues will mark this Helm release to reset the
                  values to the defaults of the targeted chart before performing an
                  upgrade. Not explicitly setting this to `false` equals to `true`
                  due to the declarative nature of the operator.
                type: boolean
              rollback:
                description: The rollback settings for this Helm release.
                properties:
                  disableHooks:
                    description: DisableHooks will mark this Helm release to prevent
                      hooks from running during the rollback.
                    type: boolean
                  enable:
                    description: Enable will mark this Helm release for rollbacks.
                    type: boolean
                  force:
                    description: Force will mark this Helm release to `--force` rollbacks.
                      This forces the resource updates through delete/recreate if
                      needed.
                    type: boolean
                  maxRetries:
                    description: MaxRetries is the maximum amount of upgrade retries
                      the operator should make before bailing.
                    format: int64
                    type: integer
                  recreate:
                    description: Recreate will mark this Helm release to `--recreate-pods`
                      for if applicable. This performs pod restarts.
                    type: boolean
                  retry:
                    description: Retry will mark this Helm release for upgrade retries
                      after a rollback.
                    type: boolean
                  timeout:
                    description: Timeout is the time to wait for any individual Kubernetes
                      operation (like Jobs for hooks) during rollback.
                    format: int64
                    type: integer
                  wait:
                    description: Wait will mark this Helm release to wait until all
                      Pods, PVCs, Services, and minimum number of Pods of a Deployment,
                      StatefulSet, or ReplicaSet are in a ready state before marking
                      the release as successful.
                    type: boolean
                type: object
              skipCRDs:
                description: SkipCRDs will mark this Helm release to skip the creation
                  of CRDs during a Helm 3 installation.
                type: boolean
              subcharts:
                items:
                  properties:
                    dependencies:
                      items:
                        type: string
                      type: array
                    name:
                      type: string
                  type: object
                type: array
              targetNamespace:
                description: TargetNamespace overrides the targeted namespace for
                  the Helm release. The default namespace equals to the namespace
                  of the HelmRelease resource.
                type: string
              test:
                description: The test settings for this Helm release.
                properties:
                  cleanup:
                    description: Cleanup, when targeting Helm 2, determines whether
                      to delete test pods between each test run initiated by the Helm
                      Operator.
                    type: boolean
                  enable:
                    description: Enable will mark this Helm release for tests.
                    type: boolean
                  ignoreFailures:
                    description: IgnoreFailures will cause a Helm release to be rolled
                      back if it fails otherwise it will be left in a released state
                    type: boolean
                  timeout:
                    description: Timeout is the time to wait for any individual Kubernetes
                      operation (like Jobs for hooks) during test.
                    format: int64
                    type: integer
                type: object
              timeout:
                description: Timeout is the time to wait for any individual Kubernetes
                  operation (like Jobs for hooks) during installation and upgrade
                  operations.
                format: int64
                type: integer
              valueFileSecrets:
                description: ValueFileSecrets holds the local name references to secrets.
                  DEPRECATED, use ValuesFrom.secretKeyRef instead.
                items:
                  properties:
                    name:
                      type: string
                  required:
                  - name
                  type: object
                type: array
              values:
                description: Values holds the values for this Helm release.
                type: object
              valuesFrom:
                items:
                  properties:
                    chartFileRef:
                      description: The reference to a local chart file with release
                        values.
                      properties:
                        optional:
                          description: Optional will mark this ChartFileSelector as
                            optional. The result of this are that operations are permitted
                            without the source, due to it e.g. being temporarily unavailable.
                          type: boolean
                        path:
                          description: Path is the file path to the source relative
                            to the chart root.
                          type: string
                      required:
                      - path
                      type: object
                    configMapKeyRef:
                      description: The reference to a config map with release values.
                      properties:
                        key:
                          type: string
                        name:
                          type: string
                        namespace:
                          type: string
                        optional:
                          type: boolean
                      required:
                      - name
                      type: object
                    externalSourceRef:
                      description: The reference to an external source with release
                        values.
                      properties:
                        optional:
                          description: Optional will mark this ExternalSourceSelector
                            as optional. The result of this are that operations are
                            permitted without the source, due to it e.g. being temporarily
                            unavailable.
                          type: boolean
                        url:
                          description: URL is the URL of the external source.
                          type: string
                      required:
                      - url
                      type: object
                    secretKeyRef:
                      description: The reference to a secret with release values.
                      properties:
                        key:
                          type: string
                        name:
                          type: string
                        namespace:
                          type: string
                        optional:
                          type: boolean
                      required:
                      - name
                      type: object
                  type: object
                type: array
              wait:
                description: Wait will mark this Helm release to wait until all Pods,
                  PVCs, Services, and minimum number of Pods of a Deployment, StatefulSet,
                  or ReplicaSet are in a ready state before marking the release as
                  successful.
                type: boolean
            required:
            - chart
            type: object
          status:
            description: ApplicationStatus defines the observed state of Application
            properties:
              application:
                description: ChartStatus denotes the current status of the Application
                  Reconciliation
                properties:
                  error:
                    type: string
                  ready:
                    type: boolean
                  staged:
                    type: boolean
                type: object
              name:
                type: string
              subcharts:
                additionalProperties:
                  description: ChartStatus denotes the current status of the Application
                    Reconciliation
                  properties:
                    error:
                      type: string
                    ready:
                      type: boolean
                    staged:
                      type: boolean
                  type: object
                type: object
            required:
            - application
            - name
            - subcharts
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

Custom Resource object

file: application-obj.yaml

# Example: Kafka with Zookeeper as a dependency
apiVersion: orkestra.azure.microsoft.com/v1alpha1
kind: Application
metadata:
  name: kafka-dev
spec:
  # Namespace to which the HelmRelease object is deployed
  namespace: "default"
  repo: bitnami
  groupID: "dev"
  subcharts:
    # subchart ordering
    - name: zookeeper
      dependencies: []
  # HelmRelease spec fields
  # https://docs.fluxcd.io/projects/helm-operator/en/1.0.0-rc9/references/helmrelease-custom-resource.html#helmrelease-custom-resource
  chart:
    repository: "https://charts.bitnami.com/bitnami"
    name: kafka
    version: 12.4.1
  values:
    global:
      imagePullSecrets: []
    zookeeper:
      enable: true
kubectl apply -f application-crd.yaml
kubectl apply -f application-obj.yaml
kubectl get applications kafka-dev -o yaml

output:

apiVersion: orkestra.azure.microsoft.com/v1alpha1
kind: Application
metadata:
  creationTimestamp: "2021-02-02T03:07:47Z"
  generation: 1
  managedFields:
  - apiVersion: orkestra.azure.microsoft.com/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:chart:
          .: {}
          f:name: {}
          f:repository: {}
          f:version: {}
        f:groupID: {}
        f:namespace: {}
        f:repo: {}
        f:subcharts: {}
        f:values: {}
    manager: kubectl-create
    operation: Update
    time: "2021-02-02T03:07:47Z"
  name: kafka-dev
  resourceVersion: "8320"
  selfLink: /apis/orkestra.azure.microsoft.com/v1alpha1/applications/kafka-dev
  uid: 92463bed-5b59-4776-8339-5277be1bda72
spec:
  chart:
    name: kafka
    repository: https://charts.bitnami.com/bitnami
    version: 12.4.1
  groupID: dev
  namespace: default
  repo: bitnami
  subcharts:
  - dependencies: []
    name: zookeeper
  values: {}

Anything else we need to know?:

The embedded field is the same as https://github.com/fluxcd/helm-operator/blob/v1.2.0/chart/helm-operator/crds/helmrelease.yaml#L237-L239 in Helm Operator project, HelmRelease object

Environment:

neolit123 commented 3 years ago

/sig api-machinery

liggitt commented 3 years ago

your schema does not define any properties in the values field:

              values:
                description: Values holds the values for this Helm release.
                type: object

By default, unknown fields are pruned. To preserve them, specify x-kubernetes-preserve-unknown-fields: true, per https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#controlling-pruning

              values:
                description: Values holds the values for this Helm release.
                type: object
                x-kubernetes-preserve-unknown-fields: true
nitishm commented 3 years ago

Thanks @liggitt that solved the problem :+1:

fedebongio commented 3 years ago

/triage accepted

fedebongio commented 3 years ago

Thank you Jordan!