search

vmware-archive / kind-on-c

Make it easy to run kind as a concourse task -- hopefully!
MIT License
24 stars 8 forks source link
concourse concourse-ci concourse-task concourse-tasks kind kubernetes test testing

readme

KINDONC!! a.k.a. kind-on-c a.k.a. kind on concourse

KINDONC!! POW!!

This is a [concourse] task which uses kind to start a kubernetes cluster locally for users to run their tests against.

Users can choose to run whichever kubernetes version kind uses by default, or use a specific kubernetes version by providing the kubernetes source tree checked out at a specific version.

It is heavily based on concourse-dcind and of course uses kind. This is really just plumbing ...

Usage

Run whatever kind thinks is cool ...

jobs:
- name: kind
  plan:
  - in_parallel:
    - get: kind-on-c
    - get: kind-release
      params:
        globs:
        - kind-linux-amd64
  - task: run-kind
    privileged: true
    file: kind-on-c/kind.yaml
    params:
      KIND_TESTS: |
        # your actual tests go here!
        kubectl get nodes -o wide

resources:
- name: kind-release
  type: github-release
  source:
    owner: kubernetes-sigs
    repository: kind
    access_token: <some github token>
    pre_release: true
- name: kind-on-c
  type: git
  source:
    uri: https://github.com/pivotal-k8s/kind-on-c

This uses the kind binary, provided by the kind-release resource, to create a kubernetes cluster. It will use the node image from the kind team. The version of kind in use and thus, indirectly, the kubernetes version deployed can be controlled by pinning the kind-release resource to a specific version.

When the cluster is up and running, any commands configured in $KIND_TESTS will run. The environment is setup with $KUBECONFIG which points to the client configuration for the deployed kubernetes cluster. Also the most recent (stable) version of kubectl will be downloaded and put into the $PATH, therefore kubectl <something> will just work™️.

Build and run your own kubernetes ...

jobs:
- name: kind
  plan:
  - in_parallel:
    - get: k8s-git
    - get: kind-on-c
    - get: kind-release
      params:
        globs:
        - kind-linux-amd64
  - task: run-kind
    privileged: true
    file: kind-on-c/kind.yaml
    params:
      KIND_TESTS: |
        # your actual tests go here!
        kubectl get nodes -o wide

resources:
- name: k8s-git
  type: git
  source:
    uri: https://github.com/kubernetes/kubernetes
- name: kind-release
  type: github-release
  source:
    owner: kubernetes-sigs
    repository: kind
    access_token: <some github token>
    pre_release: true
- name: kind-on-c
  type: git
  source:
    uri: https://github.com/pivotal-k8s/kind-on-c

If the task finds an task input named k8s-git it treats that as a kubernetes source tree and tells kind to create a node image off of that. You can just use a git resource, and pin it to a specific commit if need be, if you want to run a specific kubernetes version.

In this case, also kubectl is compiled on the fly and therefore exactly matches the version of kubernetes deployed.

User configurations

Well known task inputs & outputs

Inputs

Outputs

The task generates all the outputs in this list, however depending on the task's configuration it may leave certain (or even all) outputs empty.

Bring your own task image

Chances are good, that you need some additional tools to run your KIND_TESTS. To do that, you can use a custom task image.

You can e.g. use the registry-image resource to pull in your custom image and then override kind-on-c's default image by providing its rootfs to the kind-on-c task via the image directive. This overrides the image that is configured in the task config.

When it comes to creating the image, you can base it on the default kind-on-c image and add your additional tools and things as another layer. For that you can follow the image's ci-latest tag, which gets updated every time we push a new task image. Or you can extract all information (repo and digest) of the default task image from the task config file.

When you want to bring a entirely different image which is not based on kind-on-c's default task image, you need to check which dependencies we need in the Dockerfile.

For e.g. static binaries you could, instead of providing a custom image, have concourse pull down all the needed things and provide them to the task via the inputs input. Also, of course, nothing stand in the way to use both a custom image and provide certain things via inputs.

jobs:
- name: kind
  plan:
  - in_parallel:
    - get: custom-kind-on-c-image
    - get: kind-on-c
    - get: kind-release
    - get: other-custom-things
  - task: run-kind
    privileged: true
    file: kind-on-c/kind.yaml
    image: custom-kind-on-c-image # This overrides the image from the task config file
    input_mapping:
      inputs: other-custom-things
    params:
      KIND_TESTS: # use custom things either from the image or from inputs

resources:
- name: custom-kind-on-c-image
  type: registry-image
  source: {repository: my-custom-image, tag: "1.13"}
- name: other-custom-things
  type: some-resource-type
  source: {...}

Aggregated inputs & outputs

Currently this task only allows for a fixed set of inputs and outputs (some inputs are optional, some output might be kept empty). This set cannot be changed by users of kind-on-c.

Still users need to be able to provide one or more of their own resources to be consumed by this task as inputs or generate one or more outputs by this task.

A workaround for this are (what I call now) aggregated inputs & outputs:

Users can use as many input resources they want, they however need to aggregate them into the one input inputs for this task. Likewise, if that is something a user needs to do, they can place all their outputs into the single outputs output and have a later task split them apart again, into separate, individual outputs.

After all, inputs and outputs are eventually "just directories" on disk and therefore relatively easy to manipulate if need be.

Example usage of aggregated inputs and outputs

The following (contrived) example shows how this could be used:

plan:
- in_parallel:
  - get: kind-on-c
  - get: input-res-1
    trigger: true
  - get: input-res-2
    trigger: true
  - get: input-res-3
- task: aggregate inputs for kind-on-c
  config:
    platform: linux
    image_resource:
      type: registry-image
      source: { repository: bash }
    inputs:
    - name: input-res-1
    - name: input-res-2
    - name: input-res-3
    outputs:
    - name: inputs # this will be the aggregated inputs for kind-on-c
    run:
      path: bash
      args:
      - -xeuc
      - |
        cp -a input-res-1 inputs/
        cp -a input-res-2 inputs/
        cp -a input-res-3 inputs/
- task: kind-on-c
  privileged: true
  file: kind-on-c/kind.yaml
  params:
    KIND_TESTS: |
      # your actual tests go here!
      cp -a some/artifacts outputs
      cp -a some/metrics   outputs
      cp -a some/logs      outputs
- task: split aggregated outputs of kind-on-c
  config:
    platform: linux
    image_resource:
      type: registry-image
      source: { repository: bash }
    inputs:
    - name: outputs
    outputs:
    - name: artifacts
    - name: metrics
    - name: logs
    run:
      path: bash
      args:
      - -xeuc
      - |
        cp -a outputs/artifacts/* artifacts
        cp -a outputs/metrics/*   metrics
        cp -a outputs/logs/*      logs
- in_parallel:
  - put: artifactory          # using only the artifacts output of the last task
    params:
      folder: artifacts
  - put: lftp-log-dump        # using only the log output of the previous task
    params:
      files: "logs/*.log"
  - put: swift-metrics-store  # using only the metrics output of the previous task
    params:
      from: "metrics/(.*)"

But why?

kind-on-c made the design decision to use a task file. This is mostly to have an easy way to specify the specific version of the kind-on-c image, have the pipeline automatically update it, and track which commit has been tested with which version of the image.

concourse however does not allow to mix settings (e.g. inputs) from a task file with inline configuration in the task config.

Therefore, for now at least, kind-on-c is opting for specifying the set of allowed/available inputs/outputs, which might mean a bit of overhead for the users.