Empty info custom resource state metrics not collected

[robbie-demuth]

I'm using the custom resource state metrics feature to collect custom resource state metrics for custom resources of a custom resource definition our team owns. My config is as follows:

spec:
  resources:
    - groupVersionKind:
        group: data.appian.com
        version: v1alpha1
        kind: Operation
      metricNamePrefix: kube_operation
      labelsFromPath:
        tenant_id: [metadata, name]
        type: [spec, type]
      metrics:
        - name: info
          help: Information about operation.
          each:
            type: Info
            info: {}

What happened:

kube_operation_info metrics were not collected

$ curl -s localhost:8080/metrics | grep kube_operation_info
# HELP kube_operation_info Information about operation.
# TYPE kube_operation_info info

What you expected to happen:

kube_operation_info metrics would be collected

$ curl -s localhost:8080/metrics | grep kube_operation_info
# HELP kube_operation_info Information about operation.
# TYPE kube_operation_info info
kube_operation_info{customresource_group="data.appian.com",customresource_kind="Operation",customresource_version="v1alpha1",tenant_id="12345",type="OnDemandBackup"} 1

How to reproduce it (as minimally and precisely as possible):

Define a type: Info metric with info: {}

Anything else we need to know?:

Originally, I used the following config:

spec:
  resources:
    - groupVersionKind:
        group: data.appian.com
        version: v1alpha1
        kind: Operation
      metricNamePrefix: kube_operation
      labelsFromPath:
        tenant_id: [metadata, name]
      metrics:
        - name: info
          help: Information about operation.
          each:
            type: Info
            info:
              labelsFromPath:
                type: [spec, type]

This config worked. My config had other metrics, and I wanted the type label to be present on all of them, so I moved it from the info metric level to the resource level. That broke collection of info metrics. The following config, however, while verbose, does work:

spec:
  resources:
    - groupVersionKind:
        group: data.appian.com
        version: v1alpha1
        kind: Operation
      metricNamePrefix: kube_operation
      labelsFromPath:
        tenant_id: [metadata, name]
        type: [spec, type] # For all metrics (aside from the info metric)
      metrics:
        - name: info
          help: Information about operation.
          each:
            type: Info
            info:
              labelsFromPath:
                type: [spec, type] # For the info metric

In this case, all metrics have the type label

Environment:

• kube-state-metrics version: v2.8.2
• Kubernetes version (use kubectl version): v1.24.17-eks-4f4795d

[CatherineF-dev]

Could you give a detailed example (CRD + CR) so that I can reproduce?

You can change CRD / CR a little bit as long as it can be reproduced.

[robbie-demuth]

@CatherineF-dev, sure! Here's our CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.13.0
  name: operations.data.appian.com
spec:
  conversion:
    strategy: None
  group: data.appian.com
  names:
    kind: Operation
    listKind: OperationList
    plural: operations
    singular: operation
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - jsonPath: .spec.type
      name: Type
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Operation is the Schema for the operations API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: OperationSpec defines the desired state of Operation
            properties:
              move:
                description: Move operation
                properties:
                  region:
                    description: Region to move site to
                    minLength: 1
                    type: string
                required:
                - region
                type: object
              restore:
                description: Restore operation
                properties:
                  tenantID:
                    description: TenantID to restore site from Forbidden when spec.type
                      is not Restore
                    minLength: 1
                    type: string
                  timestamp:
                    description: Timestamp to restore site to
                    format: date-time
                    type: string
                required:
                - timestamp
                type: object
              type:
                description: Type of the operation
                enum:
                - ShutdownBackup
                - PeriodicBackup
                - OnDemandBackup
                - Create
                - Restore
                - RestoreLatest
                - Delete
                - Move
                type: string
            required:
            - type
            type: object
          status:
            description: OperationStatus defines the observed state of Operation
            properties:
              conditions:
                description: Represents the observations of an operation's current
                  state.
                items:
                  description: "Condition contains details for one aspect of the current
                    state of this API Resource. --- This struct is intended for direct
                    use as an array at the field path .status.conditions.  For example,
                    \n type FooStatus struct{ // Represents the observations of a
                    foo's current state. // Known .status.conditions.type are: \"Available\",
                    \"Progressing\", and \"Degraded\" // +patchMergeKey=type // +patchStrategy=merge
                    // +listType=map // +listMapKey=type Conditions []metav1.Condition
                    `json:\"conditions,omitempty\" patchStrategy:\"merge\" patchMergeKey:\"type\"
                    protobuf:\"bytes,1,rep,name=conditions\"` \n // other fields }"
                  properties:
                    lastTransitionTime:
                      description: lastTransitionTime is the last time the condition
                        transitioned from one status to another. This should be when
                        the underlying condition changed.  If that is not known, then
                        using the time when the API field changed is acceptable.
                      format: date-time
                      type: string
                    message:
                      description: message is a human readable message indicating
                        details about the transition. This may be an empty string.
                      maxLength: 32768
                      type: string
                    observedGeneration:
                      description: observedGeneration represents the .metadata.generation
                        that the condition was set based upon. For instance, if .metadata.generation
                        is currently 12, but the .status.conditions[x].observedGeneration
                        is 9, the condition is out of date with respect to the current
                        state of the instance.
                      format: int64
                      minimum: 0
                      type: integer
                    reason:
                      description: reason contains a programmatic identifier indicating
                        the reason for the condition's last transition. Producers
                        of specific condition types may define expected values and
                        meanings for this field, and whether the values are considered
                        a guaranteed API. The value should be a CamelCase string.
                        This field may not be empty.
                      maxLength: 1024
                      minLength: 1
                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
                      type: string
                    status:
                      description: status of the condition, one of True, False, Unknown.
                      enum:
                      - "True"
                      - "False"
                      - Unknown
                      type: string
                    type:
                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
                        --- Many .condition.type values are consistent across resources
                        like Available, but because arbitrary conditions can be useful
                        (see .node.status.conditions), the ability to deconflict is
                        important. The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
                      maxLength: 316
                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
                      type: string
                  required:
                  - lastTransitionTime
                  - message
                  - reason
                  - status
                  - type
                  type: object
                type: array
                x-kubernetes-list-map-keys:
                - type
                x-kubernetes-list-type: map
              results:
                additionalProperties:
                  description: HandlerResult will indicate if the handler has succeeded
                    or failed
                  enum:
                  - Succeeded
                  - Failed
                  type: string
                type: object
            type: object
        required:
        - spec
        type: object
    served: true
    storage: true
    subresources:
      status: {}

And here's an example CR:

apiVersion: data.appian.com/v1alpha1
kind: Operation
metadata:
  name: "12345"
spec:
  type: OnDemandBackup

[dgrisonnet]

/assign @rexagod /triage accepted

[prakrit55]

@dgrisonnet, @rexagod I wd like to give it a try 😄

[dgrisonnet]

Awesome! Feel free to reach out to any of us if you need any help along the way :)

[prakrit55]

Awesome! Feel free to reach out to any of us if you need any help along the way :)

@dgrisonnet, thanks for the assignment😃, definitely going to ping you guys in slack if required