search

kcl-lang / kcl

KCL Programming Language (CNCF Sandbox Project). https://kcl-lang.io
https://kcl-lang.io
Apache License 2.0
1.45k stars 109 forks source link

[Feature][Design] KRM KCL Spec & User Story #479

Closed Peefy closed 8 months ago

Peefy commented 1 year ago

Introduction

KCL is specifically used for configuration writing and policy validation in cloud-native and Kubernetes scenarios. However, it is obviously not enough to use only one language for Kubernetes configuration management. KCL needs to be integrated with more scenarios and specifications to make it easier and more standardized for users to use KCL in their specific scenarios.

KCL was initially used for modeling, and Kubernetes naturally provided the KRM model. We can reuse existing resources and standardize them in KCL's registry using tools such as KCL OpenAPI and KPM. For example, users can reuse existing resources and abstract their own models (building models such as containers, services, computing, storage, and networks) and reuse them.

Overview

This issue is mainly used to clarify user stories and unified specifications of various KCL scenario models mainly for KRM. (👀 TBD: Perhaps we can give it a better name instead of KRM KCL Spec...)

Pain Points

See Appendix for more

User Story

User Story I (Day1 Deployment)

As SRE/Ops (who), ... I want (what), because (why) ...

I hope to achieve fast and reliable application deployment and updates through deployment automation, improving deployment efficiency, and reducing error rates. Specifically, I hope to have a simple and easy-to-use automation deployment tool that can help me complete the following tasks:

I hope this automation deployment tool can support various application programming languages and frameworks, as well as multiple deployment targets such as virtual machines, containers, and cloud platforms. Additionally, I hope this tool can be integrated with other operations and maintenance tools to achieve a comprehensive automation operations and maintenance chain.

User Story II (Day2 Operation)

As SRE/Ops (who), ... I want (what), because (why) ...

Here are some examples of Day2 operations for SRE:

User Story III

As application developers (who), ... I want (what), because (why) ...

The most important concern for application developers might be to devote maximum their attention to the functional-related development, and to shorter the development-validation cycles, which bring about continuous progressive product enhancement.

User Story IV

As platform developers (who), ... I want (what), because (why) ...

Solution

Design

KRM KCL Spec

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  annotations: 
    krm.kcl.dev/version: 0.0.1
    krm.kcl.dev/type: abstraction # abstraction, mutation, validation
    documentation: >-
        KRM KCL Documents
# EDIT THE SOURCE!
# This should be your KCL code which preloads the `ResourceList` to `option("resource_list").
spec:
  # code source
  params:
     param1: param_value1
     param2: param_value2
  source: |
    # Add annotations
    [resource | {if resource.kind == "Deployment": metadata.annotations: {"managed-by" = "helmfile-kcl"}} for resource in option("resource_list").items]
  # source: path/to/file
  # source: oci://
  # source: git://
  # source: https://

The above specifications are the same for any cloud-native tool and K8s controller that integrates KCL (e.g., Helm KCL Plugin, Kustomize KCL Plugin). The usage of the specifications can be mainly divided into three categories: mutation, validation, and abstraction. Through KRM-KCL, we can generate YAML using KCL, and perform mutation or validation based on the stock of YAML.

KRM KCL Function Spec

A KRM KCL function contains as follows

image

where:

The spec definition (WIP: v1alpha1)

schema KCLRun:
    apiVersion: "krm.kcl.dev/v1alpha1" | "krm.kcl.dev/v1beta1" | "krm.kcl.dev/v1" = "krm.kcl.dev/v1alpha1" # The resource API version
    kind: "KCLRun" = "KCLRun" # The resource type
    spec: KCLRunSpec # The resource spec
    status?: KCLRunStatus # The resource status

schema KCLRunSpec:
    source: str # KCL code source, which can come from local files, git repositories, or OCI source code
    params: {str:} # Params: Input parameters of the model.

schema KCLRunStatus:
    message: str  # Diagnostic message for running KCL.
    severity: "error" | "warning" | "info"  # Diagnostic type for running KCL.
    resourceRef?: ResourceRef  # Which KRM and KCL execution encountered problems?

schema ResourceRef:
    apiVersion: str
    kind: str
    name: str
    namespace: str

Examples

apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: set-annotations
  metadata:
    annotations: 
      krm.kcl.dev/version: 0.0.1
      krm.kcl.dev/type: mutation
      documentation: >-
        Add or change annotations
spec:
  params:
    toAdd: addValue
  source: oci://ghcr.io/kcl-lang/set-annotation
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: https-only
  metadata:
    annotations: 
      krm.kcl.dev/version: 0.0.1
      krm.kcl.dev/type: validation
      documentation: >-
        Requires Ingress resources to be HTTPS only.  Ingress resources must
        include the `kubernetes.io/ingress.allow-http` annotation, set to `false`.
        By default a valid TLS {} configuration is required, this can be made
        optional by setting the `tlsOptional` parameter to `true`.
        More info: https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
spec:
  source: oci://ghcr.io/kcl-lang/https-only
apiVersion: krm.kcl.dev/v1alpha1
kind: KCLRun
metadata:
  name: web-service
  metadata:
    annotations: 
      krm.kcl.dev/version: 0.0.1
      krm.kcl.dev/type: abstraction
      documentation: >-
        Web service application abstraction
spec:
  params:
    name: app
    containers:
      ngnix:
        image: ngnix
        ports:
          containerPort: 80
    labels:
      name: app
  source: oci://ghcr.io/kcl-lang/web-service

The output is

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    name: app
spec:
  selector:
    matchLabels:
      name: app
  template:
    metadata:
      labels:
        name: app
    spec:
      containers:
        - name: ngnix
          image: ngnix
          ports:
            containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: app
  labels:
    name: app
spec:
  selector:
    name: app
  ports:
    - port: 80

Components

image

image

How to (WIP)

Use IDE products and CLI tools such as KPM as a unified interface for user interaction. At the basic feature level, CLI/IDE/Dashboard is equivalent

image

Take CLI as an example

Run

Code

Doc

Check

Lint
Test

Import

Convert various data/DSL configurations and schemas of inventory into KCL

image

KRM KCL Implementation

KRM KCL Examples

https://github.com/kcl-lang/krm-kcl/tree/main/examples/mutation

https://github.com/kcl-lang/krm-kcl/tree/main/examples/validation

https://github.com/kcl-lang/krm-kcl/tree/main/examples/abstraction

Further More

Automation & GitOps

Moreover, the things defined in the above specifications can still be used in Dev-CI-CD and other operation and maintenance automation processes in combination with IDE and other products, such as overall integration: Gitlab (Dev) - Jenkins (CI) - Harbor (Registry) - KRM-KCL Spec (Config) - ArgoCD (CD) - Kubernetes (Runtime).

image

image

Config

Regardless of the configuration method used, they are all friendly to the operation and maintenance automation process.

Data/Schema Migration

Appendix

Real feedback

image image

image

image

image

image

Reference

Peefy commented 1 year ago

@ekkoful Hi, here. I hope we can participate in improving this content together. ❤️ Ultimately, we use a unified specification to support client and runtime KCL integration tools.

ekkoful commented 1 year ago

@ekkoful Hi, here. I hope we can participate in improving this content together. heart Ultimately, we use a unified specification to support client and runtime KCL integration tools.

Yeah. Good job. It's a good arch pic for developer. image

ekkoful commented 1 year ago
image

I plan to implement the KCLRun kind resource for kubernetes resource. There is the basic design. cc: @Peefy

Note: The KCLRun kind maybe update the kubernetes resource, like pod, deployment...

Peefy commented 1 year ago
image

I plan to implement the KCLRun kind resource for kubernetes resource. There is the basic design. cc: @Peefy

Note: The KCLRun kind maybe update the kubernetes resource, like pod, deployment...

OK, no problem!

Peefy commented 12 months ago
image

I plan to implement the KCLRun kind resource for kubernetes resource. There is the basic design. cc: @Peefy Note: The KCLRun kind maybe update the kubernetes resource, like pod, deployment...

OK, no problem!

What needs to be added is: