search

kubernetes-sigs / application

Application metadata descriptor CRD
Apache License 2.0
511 stars 162 forks source link

Application Readiness #78

Closed kow3ns closed 5 years ago

kow3ns commented 5 years ago

Application Status

Objective

The objective of this proposal is to provide a mechanism to aggregate the status of an Application. We propose a mechanism to compute the readiness, availability, errors, and disruptions associated with an Application. As black and white box health monitoring are a complicated topic that deserves its own treatment, we do not address it in this proposal.

Background

Conditions

Conditions are used across the Kubernetes API surface in order to indicate the condition of a resource as its controllerseeks to realize the declared intent in its specification. They are described by the golang struct below. Throughout this proposal we use conditions in conjunction with fields.

type Condition struct {
    // Type is the type of the condition.
    Type ComponentConditionType `json:"type"`
    // Status is one of True, False, Unknown.
    Status v1.ConditionStatus `json:"status"`
    // LastTransitionTime is the last time the condition's status changed.
    LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
    // Reason is the reason for the condition's last transition.
    Reason string `json:"reason,omitempty"`
    // Message is a human readable message indicating details about the transition.
    Message string `json:"message,omitempty"`
}

Readiness

From the perspective of Pods, readiness indicates the ability of the Pod to receive network traffic. It also indicates that the resource, from the perspective of the control loops that act on it, is ready for use. We use the same semantics for the components of an Application. Readiness for an Application implies that all of its components are ready. That is, an Application is ready if and only if all of its components are ready. Components that contain no user declared desired state (i.e. have no spec) (e.g. ConfigMaps and Secrets) are always ready post creation.

  1. A controller MAY communicate that a resource is ready by indicating status.ready=true or by including a condition like {"type":"Ready","status":"true"} in the resource's status.conditions.
  2. A controller MAY communicate that a resource is unready by indicating status.ready=false or by including a condition like {"type":"Ready","status":"false"} in the resource's status.conditions.
  3. If a controller communicates conflicting readiness (e.g setting status.ready=true and including a condition like {"type":"Ready","status":"false"}) the value of the field takes precedence.
  4. If a controller uses both a status field and a condition to communicate readiness, and if the field and condition are consistent, the condition is treated as a decorator for the field (e.g settingstatus.ready=true and including a condition like {"type":"Ready","status":"false","message":"RDBMs is ready for use."} provides a user readable message about the readiness of the resource).

Availability

From the perspective of Deployments, and many out of tree resources, availability, indicates that all Pods in the related to the resource remain ready after some configurable duration. This notion is useful for application components in general as indication that the resource is unlikely to fall victim to infant mortality after creation or mutation. We use availability for an Application's components in this context. As availability is not applicable to all components, it is not aggregated for an Application.

  1. A controller MAY communicate that a resource is available by indicating status.available=true or by including a condition like {"type":"Available","status":"true"} in the resource's status.conditions.
  2. A controller MAY communicate that a resource is unavailable by indicating status.available=false or by including a condition like {"type":"Available","status":"false"} in the resource's status.conditions.
  3. If a controller communicates conflicting availability (e.g setting status.availabile=true and including a condition like {"type":"Available","status":"false"}) the value of the field takes precedence.
  4. If a controller uses both a status field and a condition to communicate availability, and if the field and condition are consistent, the condition is treated as a decorator for the field.

Observation

Kubernetes control loops communicate that they have observed modifications of the declared desired state contained in a resource by setting its status.observedGeneration to its generation. A resource for which this is true is said to be observed.

  1. Controllers SHOULD update status.observedGeneration to the value of meta.generantion to communicate that they have observed the creation of, or a mutation to, a resource they control.

Progress

Kubernetes control loop use various methods to communicate that reconciliation between a resources specification and the observed state of the system is progressing. Deployments, and many non-core resources, communicate this using the Progressing condition. For an Application, progressing components indicate that the application is updating.

  1. A controller MAY communicate that reconciliation is in progress by indicating setting status.progress to a true or to a non-negative 32-bit floating point number between in [0,100] (e.g status.progress=true or status.progress=99.9).
  2. A controller MAY communicate that reconciliation is in including a condition like {"type":"Progressing","status":"true"} in the resource's status.conditions.
  3. If a controller communicates conflicting progress (e.g setting status.progressing=true and including a condition like {"type":"Progressing","status":"false"}) the value of the field takes precedence.
  4. If a controller uses both a status field and a condition to communicate readiness, and if the field and condition are consistent, the condition is treated as a decorator for the field.

Disruptions

Application components may be affected by planned or unplanned disruptions. For instance the destruction of a Node may disrupt many replicated Pod sets. The application controller MAY use other resources, e.g PodDisruptionBudgets, to add this condition to a resource's status, and the controller for a resource MAY communicate this directly by adding such a condition.

  1. A controller MAY communicate disruptions using by adding a condition like {"type":"Disruption","status":"true","reason":"Node unavailable","message":"Auto-scaling in progress"}.
  2. Disruptions are considered to be orthogonal to readiness for the computation of Application readiness. If a disruption makes a resource unready the resources controller MUST communicate this via readiness.

Errors

At any point in their lifetime controller may encounter errors when realizing the declared intent of the user. The are communicating using the status.conditions of the resource.

  1. A controller MAY communicate an error by including a condition like {"type":"Error","status":"true","reason":"Controller Wedged","message":"SharedInformer sycn failing."}.
  2. Errors are considered to be orthogonal to readiness for the computation of Application readiness. If an error makes a resource unready the resources controller MUST communicate this via readiness.

Compatibility Requirements

Resources and controllers that wish to be compatible with the Application Controller status computation need only implement the following.

  1. Any resource that does not implement a .spec contains no declarative intent. It is always ready.
  2. The representative resource MUST implement a .status field.
  3. The .status field MUST indicate readiness.
  4. The .status field MAY indicate availability.
  5. The .status field SHOULD indicate observation by the controller.
  6. The .status field MAY contain conditions.
  7. Errors MAY be reported using error conditions in the status.conditions field.
  8. Disruptions MAY be reported using disruption conditions in the status.conditions field.

Core Resource Adaptations

The core resources do not all conform to the schema above. In the future, we may modify them to do so. For the time being, the following describes how the Application controller will compute the status of these resources.

  1. Deployment
    1. Readiness - .spec.replicas is equal to .status.readyReplicas is equal to .status.replicas and all are greater than zero.
    2. Availability - .status.conditions contains an Available condition.
    3. Progress - true if spec.conditions contains a Progressing condition.
    4. Observed - when .status.observedGeneration is equal to spec.generation.
    5. Errors - Failure conditions are converted to Error conditions.
  2. ReplicaSet and ReplicationController
    1. You should not use these directly in an Application. Deployment should be used instead.
    2. Readiness - .spec.replicas is equal to .status.replicas and .status.readyReplicas and all are greater than zero.
    3. Availability - .spec.replicas is equal to .status.replicas and .status.availableReplicas and all are greater than zero.
    4. Progress - true if .spec.replicas is not equal to .status.replicas.
    5. Observed - when .status.obloadbalancerservedGeneration is equal to spec.generation.
    6. Errors - ReplicaFailure conditions will be converted to Error conditions.
  3. StatefulSet
    1. Readiness - .spec.replicas is equal to .status.replicas and .status.readyReplicas and all are greater than zero.
    2. Availability - N/A, StatefulSet Pods are available when ready.
    3. Progress - true if .status.currentReplicas is not equal to status.updateReplicas or if .status.replicas is not equal to .spec.replicas.
    4. Observed - when .status.observedGeneration is equal to spec.generation.
    5. Errors - N/A
  4. DaemonSet
    1. Readiness - .status.currentNumberScheduled is equal to .status.desiredNumberScheduled and .status.numberReady and all are greater than 0.
    2. Availability - if .status.currentNumberScheduled is equal to .status.desiredNumberScheduled and .status.numberAvailable and all are greater than 0.
    3. Progress - DaemonSet is sensitive to Node cardinality. It is making progress when status.numberUpdated is greater than 0.
    4. Observed - when .status.observedGeneration is equal to spec.generation.
    5. Errors - N/A
  5. Pod
    1. Readiness - The Pods has a Ready condition.
    2. Availability - N/A
    3. Progress - N/A
    4. Observed - N/A
    5. Errors - N/A
  6. Ingress
    1. Readiness - .status.loadbalancer.ingress list is not empty.
    2. Availability - N/A
    3. Progress - N/A
    4. Observed - N/A
    5. Errors - N/A
  7. Service
    1. Readiness - All non-load balanced Services are ready when created. All load balanced Services are ready when .status.loadbalancer.ingerss is not empty.
    2. Availability - N/A
    3. Progress - N/A
    4. Observed - N/A
    5. Errors - N/A
  8. PersistentVolumeClaim
    1. A PersistentVolumeClaim claim is Ready when its .status.phase is Bound. This may seem strange as PVC implements a Ready phase, but a PVC is not useful to the application until it is bound, and most errors post creation and during binding.
    2. Availability - N/A
    3. Progress - N/A
    4. Observed - N/A
    5. Errors - N/A

API

This section contains the proposed modifications to the API. Here, we modify the ApplicationStatus type to report the observed status of its components. Each ComponetStatus contains a link, resource identifying information, and the ComponentConditions of the components indicated by the Application's .Spec.ComponentKinds and .Spec.Selector. The status of the applications components is used to compute ApplicationConditions that apply to the application as a whole.

// ComponentConditionType indicates the type of a ComponentCondition.
type ComponentConditionType string

const (
    // ComponentAvailable indicates that the component is available. This is used by controller to indicate that the 
    // resource has been ready for a sufficient period of time after creation or mutation that it is unlikely to suffer 
    // from infant mortality.
    ComponentAvailable ComponentConditionType = "Available"
    // ComponentReady indicates that component is ready to use.
    ComponentReady = "Ready"
    // ComponentProgressing is used to communicate that a component is in updating (i.e. A Deployment with a Rollout in
    // progress or a StatefulSet with a RollingUpdate in progress).
    ComponentProgressing = "Progressing"
    // ComponentDisrupted is used to indicate that the component is affected by a planned or unplanned disruption.
    ComponentDisrupted = "Disrupted"
    // Error is used to communicate and error condition for a component.
    ComponentError = "Error"
)

// ComponentCondition represents the condition of a component of an application. It is modeled after the Conditions
// use for the Kubernetes workloads objects.
type ComponentCondition struct {
    // Type is the type of the condition.
    Type ComponentConditionType `json:"type"`
    // Status is one of True, False, Unknown.
    Status v1.ConditionStatus `json:"status"`
    // LastTransitionTime is the last time the condition's status changed.
    LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
    // Reason is the reason for the condition's last transition.
    Reason string `json:"reason,omitempty"`
    // Message is a human readable message indicating details about the transition.
    Message string `json:"message,omitempty"`
}

// ComponentStatus contains the status of a generic component selected by the Application.
type ComponentStatus struct {
    // Name is the name of the component to which the condition pertains.
    Name string `json:"name"` ComponentConditionTyp
    // Namespace is the name containing the component to which the condition pertains.
    Namespace string `json:"namespace,omitempty"`
    // GroupKind indicates the group and kind of the component.
    GroupKind metav1.GroupKind `json:groupKind"`
    // Link is a link to the resource that represents the component.
    Link string `json:"link,omitempty"`
    // Available indicates that the component is available. This is used by controller to indicate that the 
    // resource has been ready for a sufficient period of time after creation or mutation that it is unlikely to suffer 
    // from infant mortality.
    Available *bool `json:"available,omitempty"`
    // tReady indicates that component is ready to use.
    Ready *bool `json:"ready,omitempty"`
    // Progressing is used to communicate that a component is in updating (i.e. A Deployment with a Rollout in 
    // progress or a StatefulSet with a RollingUpdate in progress).
    Progressing *bool `json:"progressing,bool"`
    // Observed indicates that the controller for the component's resource has observed the current generation.
    // (i.e .Meta.Generation == .Status.ObservedGeneration)
    Observed bool = `json:"observed,omitempty"`
    // Conditions contains the conditions that are applicable to the component.
    Conditions [] ComponentCondition `json:"conditions,omitempty"`
}

// ApplicationConditionType indicates the type of an application condition.
type ApplicationConditionType string

const (
    // ApplicationError is used to communicate that an Application has an Error condition.
    ApplicationError ApplicationConditionType = "Error"
)

// ApplicationCondition represents the condition of an Application.
type ApplicationCondition struct {
    // Type is the type of the condition.
    Type ApplicationConditionType `json:"type"`
    // Status is one of True, False, Unknown.
    Status v1.ConditionStatus `json:"status"`
    // LastTransitionTime is the last time the condition's status changed.
    LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
    // Reason is the reason for the condition's last transition.
    Reason string `json:"reason,omitempty"`
    // Message is a human readable message indicating details about the transition.
    Message string `json:"message,omitempty"`
}

type ApplicationStatus struct {
    // Ready indicates that all of an Application's components are Available.
    Ready *bool `json:"available,omitempty"`
    // Updating is used to communicate that a component is in updating (i.e. A Deployment with a Rollout in 
    // progress or a StatefulSet with a RollingUpdate in progress. 
    Updating *bool `json:"progressing,bool"`
    // Conditions is a list of ApplicationConditions for the application.
    Conditions [] ApplicationCondition
    //ComponentStatus is a list of the statues of the applications components.
    Components [] ComponentStatus
}

Application Status Computation

The Application controller will periodically list the applications residing on the API Server. For each Application resource the controller will do the following.

  1. If the spec.assemblyPhase of the Application is pending the controller will not update the .status of the Application. This allows application installers time apply all necessary components prior to application status computation.
  2. If the Application is assembled, the controller will do the following for each GroupKind indicated by the Application's .spec.componentKinds.
    1. Use the discovery API to get the default version of the resource.
    2. Retrieve the resource from the API Server.
    3. Determine the readiness, availability, progress and observed status of each component with respect to the core resource adaptations and use this create a corresponding ComponentStatus and to append it to the .status.components of the Application.
    4. If one or more components of the Application are Progressing the status.updating field is set to true.
    5. If the Application is assembled and all of its components are ready the status.ready field is set to true.
    6. If the Application is assembled and any of its components are not ready the status.ready field is set to false.
  3. The Application controller will then update the Application to communicate the status to the end user.

Example


apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
  name: "wordpress-01"
  labels:
    app.kubernetes.io/name: "wordpress-01"
spec:
  type: "wordpress"
  selector:
    matchLabels:
     app.kubernetes.io/name: "wordpress-01"
  componentKinds:
    - group: core
      kind: Service
    - group: apps
      kind: StatefulSet
  version: "4.9.4"
  description: "WordPress is open source software you can use to create a beautiful website, blog, or app."
  icons:
    - src: "https://s.w.org/style/images/about/WordPress-logotype-wmark.png"
      type: "image/png"
      size: "1000x1000"
    - src: "https://s.w.org/style/images/about/WordPress-logotype-standard.png"
      type: "image/png"
      size: "2000x680"
  maintainers:
    - name: Kenneth Owens
      email: kow3ns@github.com
  owners:
    - name: Kenneth Owens
      email: kow3ns@github.com
  keywords:
   - "cms"
   - "blog"
   - "wordpress"
  links:
    - description: About
      url: "https://wordpress.org/"
    - description: Web Server Dashboard
      url: "https://metrics/internal/wordpress-01/web-app"
    - description: Mysql Dashboard
      url: "https://metrics/internal/wordpress-01/mysql"
status:
  ready: true
  updating: false
  components:
    - name: wordpress-mysql-hvc
      namespace: default
      groupKind: Service
      link: /apis/v1/namespaces/default/services/wordpress-mysql-hvc
      ready: true
    - name: wordpress-mysql
      namespace: default
      groupKind: apps/StatefulSet
      link: /apis/apps/v1/namespaces/default/statefulsets/wordpress-mysql
      ready: true
      avialable: true
    - name: wordpress-webserver-svc
      namespace: default
      groupKind: Service
      link: /apis/v1/namespaces/default/services/wordpress-webserver-svc
      ready: true
    - name: wordpress-webserver
      namespace: default
      groupKind: apps/StatefulSet
      link: /apis/apps/v1/namespaces/default/statefulsets/wordpress-webserver
      ready: true
      avialable: true
ant31 commented 5 years ago

@kow3ns similarly to : https://github.com/kubernetes-sigs/application/issues/76#issuecomment-435105857, could you capture this in a PR/.md, so it's easier to comment?

barney-s commented 5 years ago

Hey @kow3ns Should we abandon #76 and focus on this one ? This is a more fleshed out version of #76 (meaning #76 could be enhanced to become a version of this).

konryd commented 5 years ago

Please consider specifying the behavior for nested Apps. Either explicitly saying it will not have a status as a component or handle the (nasty) implications in the doc.

Admittedly, it is not a core component, but one that the App CRD itself could not pretend not to be aware of :)

erictune commented 5 years ago

I want to propose changes to this issue. They are longer than can reasonably fit in a comment. What is the suggested way to do that? New issue or google doc or PR?

ant31 commented 5 years ago

I think google doc is easier to iterate and comment

fejta-bot commented 5 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 5 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 5 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 5 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/application/issues/78#issuecomment-505602816): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.