search

containerbuilding / cbi

Container Builder Interface for Kubernetes with support for several backends (Docker, BuildKit, Buildah, kaniko, img, Google Cloud Container Builder, Azure Container Registry Build, OpenShift Source-to-Image...)
Apache License 2.0
243 stars 18 forks source link
buildah buildkit docker docker-image dockerfile kaniko kubernetes opencontainers

readme

:warning: CBI is no longer under active development, in favor of Tekton. See tektoncd/pipeline and tektoncd/catalog.

CBI: Container Builder Interface for Kubernetes

CBI provides a vendor-neutral interface for building (and pushing) container images on top of a Kubernetes cluster, with support for several backends such as Docker, BuildKit, Buildah, kaniko, img, Google Cloud Container Builder, Azure Container Registry Build and OpenShift Source-to-Image (S2I).

cbi.png

Current status

Specification

Implementation

Plugin Backend Dockerfile cloudbuild.yaml OpenShift S2I BuildKit LLB ACB Pipeline
docker Docker Yes ✅
buildkit BuildKit Yes ✅ Planned
buildah Buildah Yes ✅
kaniko kaniko Yes ✅
img img Yes ✅
gcb Google Cloud Container Builder Yes ✅ Yes ✅
acb Azure Container Registry Build Yes ✅ Planned Planned
s2i OpenShift Source-to-Image (S2I) Yes ✅

Please feel free to open PRs to add other plugins.

Quick start

Requires Kubernetes 1.9 or later.

Installation

$ kubectl apply -f https://raw.githubusercontent.com/containerbuilding/cbi/master/cbi-latest.yaml

Everything (except CustomResourceDefinition, ClusterRole, and ClusterRoleBinding) will be installed to the cbi-system namespace.

You may edit the YAML file to change the namespace or to add NetworkPolicy.

Hint for Google Kubernetes Engine (GKE) users

If you hit an error while creating clusterrolebindings, you need to execute kubectl command as follows: 

$ pw=$(gcloud --format json container clusters describe ${YOUR_GKE_CLUSTER_NAME} | jq -r .masterAuth.password)
$ kubectl --username=admin --password=${pw} ...

Hint for OpenShift users

oc login -u system:admin -n default might be needed before running kubectl. You would also need to enable privileged containers for most plugins. (how?)

The CBI controller daemon and the following plugins will be installed:

Plugin Requirements
docker Docker needs to be installed on the hosts
buildah Privileged containers needs to be enabled
buildKit Privileged containers needs to be enabled
kaniko None (Google Cloud is not needed)
img Privileged containers needs to be enabled (See kubernetes/community#1934 and Jess's blog for the ongoing work to remove this requirement)
gcb Requires Google Cloud service account with IAM roles, see this section (Your cluster does not need to be GKE or on GCE)
acb Requires Azure service principal with IAM roles, see this section (Your cluster does not need to be AKS or on Azure VMs)
s2i Docker needs to be installed on the hosts (OpenShift is not needed)

The default plugin is docker.

You may edit the YAML file to remove unneeded plugins or change the priorities.

Run your first buildjob

Create a buildjob ex-git-nopush from examples/ex-git-nopush.yaml:

$ kubectl create -f https://raw.githubusercontent.com/containerbuilding/cbi/master/examples/ex-git-nopush.yaml
buildjob "ex-git-nopush" created

Make sure the buildjob is created:

$ kubectl get buildjobs
NAME      AGE
ex-git-nopush       3s

Inspect the underlying job and the result:

$ kubectl get job $(kubectl get buildjob ex-git-nopush --output=jsonpath={.status.job})
NAME      DESIRED   SUCCESSFUL   AGE
ex-git-nopush-job   1         1            30s
$ kubectl logs $(kubectl get pods --selector=job-name=ex-git-nopush-job --show-all --output=jsonpath={.items..metadata.name})
Sending build context to Docker daemon 79.87 kB
Step 1 : FROM alpine:latest
...
Successfully built bef4a548fb02

Delete the buildjob (and the underlying job)

$ kubectl delete buildjobs ex-git-nopush
buildjob "ex-git-nopush" deleted

Advanced usage

Push to a registry

First you need to create a credential using kubectl create secret docker-registry .... See Kubernetes's manual.

Hint for Amazon Elastic Container Registry (ECR) users

You can create the credential as follows (expires per 12 hours): 

$ TOKEN=`aws ecr get-authorization-token --output text --query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`
$ kubectl create secret docker-registry my-registry-secret \
 --docker-server=https://12345678.dkr.ecr.ap-northeast-1.amazonaws.com \
 --docker-username=AWS \
 --docker-password="${TOKEN}" \
 --docker-email="${EMAIL}"

Hint for Google Container Registry (GCR) users

See here for creating the credential.

Hint for Azure Container Registry (ACR) users

See here for creating the credential.

You can specify the registry credential via spec.registry.secretRef.name.

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-push
spec:
  registry:
    target: example.com/foo/bar:baz
    push: true
    secretRef:
      name: docker-registry-secret-name
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
      url: ssh://me@git.example.com/foo/bar.git

Note: for Google Cloud Container Builder plugin, please refer to the Google Cloud Container Builder plugin section.

Note: for Azure Container Registry Build plugin, please refer to the Azure Container Registry Build plugin section.

Build contexts

ConfigMap context

This is the easiest but only suitable for trivial images.

Example manifest:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ex-configmap-nopush-configmap
data:
  Dockerfile: |-
    FROM busybox
    ADD hello /
    RUN cat /hello
  hello: "hello, world"
---
apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-configmap-nopush
spec:
  registry:
    target: example.com/foo/ex-configmap-nopush
    push: false
  language:
    kind: Dockerfile
  context:
    kind: ConfigMap
    configMapRef:
      name: ex-configmap-nopush-configmap

Git context

Git context is suitable for most cases.

If the repo is private, you need to create a SSH secret as follows, and specify the secret via spec.context.git.sshSecretRef.name:

$ kubectl create secret generic ssh-secret-name --from-file=id_rsa=$HOME/.ssh/id_rsa --from-file=config=$HOME/.ssh/config --from-file=known_hosts=$HOME/.ssh/known_hosts

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-nopush
spec:
  registry:
    target: example.com/foo/bar:baz
    push: false
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
# only url is mandatory. See git(1) for url spec.
      url: ssh://me@git.example.com/foo/bar.git
      revision: master
      sshSecretRef:
        name: ssh-secret-name

HTTP(S) context

HTTP(S) context provider allows using tar(.gz) archive as a build context. This is useful for sending large contexts without interacting with a git repo.

You can create a temporary HTTP server in the Kubernetes cluster, and upload a context tarball as follows.

$ kubectl run nginx --image nginx:alpine --port 80
$ kubectl expose deployment nginx
$ tar cvf a.tar /path/to/your-context-directory
$ kubectl cp a.tar $(kubectl get pod -l run=nginx -o jsonpath={..metadata.name}):/usr/share/nginx/html

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex
spec:
  registry:
    target: example.com/foo/bar:baz
    push: false
  language:
    kind: Dockerfile
  context:
    kind: HTTP
    http:
      url: http://nginx/a.tar

Rclone context (S3, Dropbox, SFTP, and many)

Rclone supports fetching files and directories from various storage services: Amazon Drive, Amazon S3, Backblaze B2, Box, Ceph, DigitalOcean Spaces, Dreamhost, Dropbox, FTP, Google Cloud Storage, Google Drive, HTTP, Hubic, IBM COS S3, Memset Memstore, Microsoft Azure Blob Storage, Microsoft OneDrive, Minio, Nextloud, OVH, Openstack Swift, Oracle Cloud Storage, Ownloud, pCloud, put.io, QingStor, Rackspace Cloud Files, SFTP, Wasabi, WebDAV, Yandex Disk.

Any backend supported by Rclone should work with CBI, although only Amazon S3 is tested with CBI currently.

To use Rclone context provider, you need to create a secret from your ~/.config/rclone/rclone.conf, and specify as spec.context.rclone.secretRef.

$ kubectl create secret generic my-rclone-secret --from-file=$HOME/.config/rclone/rclone.conf

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-rclone-nopush
spec:
  registry:
    target: example.com/foo/ex-rclone-nopush
    push: false
  language:
    kind: Dockerfile
  context:
    kind: Rclone
    rclone:
      remote: s3
      path: my-s3-bucket/some-directory
      secretRef:
        name: my-rclone-secret

To use SFTP remote, you might need to specify spec.context.rclone.sshSecretRef as in Git context.

Plugin

Specify the plugin explicitly

Usually. the plugin is automatically selected by the CBI controller daemon.

However, if you prefer a specific plugin for some reason such as performance or stability, you can specify the plugin explicitly using spec.pluginSelector constraint.

e.g. for Buildah plugin (plugin.name=buildah),

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  ...
spec:
  pluginSelector: plugin.name=buildah
  ...

Google Cloud Container Builder plugin

You need to create a Google Cloud service account JSON with the following IAM roles in https://console.cloud.google.com/iam-admin/serviceaccounts :

And create a corresponding Kubernetes secret that contains json as follows:

$ kubectl create secret generic my-gcb --from-file=json=my-gcb-service-account.json

You don't need to use GKE (of course you can use though).

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-push
  annotations:
    cbi-gcb/secret: my-gcb
    cbi-gcb/project: my-gcp-project
spec:
  registry:
    target: gcr.io/example/foo
    push: true
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
      url: https://git.example.com/foo/bar.git
  pluginSelector: plugin.name=gcb

Note:

In addition to Dockerfile, gcb plugin also supports building images from cloudbuild.yaml.

See examples/ex-google-cloudbuild-push.yaml.sh.

Azure Container Registry Build plugin

You need to create a Azure service principal with a PEM/DER cert:

$ az ad sp create-for-rbac --name ServicePrincipalName --create-cert
{
  "appId": "...",
  "displayName": "..."",
  "fileWithCertAndPrivateKey": "my-acb.pem",
  "name": "http://...",
  "password": null,
  "tenant": "..."
}

And create a corresponding Kubernetes secret that contains cert as follows:

$ kubectl create secret generic my-acb --from-file=cert=my-acb.pem

You don't need to use AKS (of course you can use though).

Example manifest:

apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
  name: ex-git-push
  annotations:
    cbi-acb/secret: my-acb
    cbi-acb/app-id: APP_ID
    cbi-acb/tenant: TENANT
spec:
  registry:
    target: example.azurecr.io/example/foo
    push: true
  language:
    kind: Dockerfile
  context:
    kind: Git
    git:
      url: https://git.example.com/foo/bar.git
  pluginSelector: plugin.name=acb

Note:

Openshift Source-to-Image plugin

s2i plugin supports building images from S2I source but it does not support Dockerfile.

See examples/ex-s2i-nopush.yaml.

Design (subject to change)

Components

CBI is composed of the following specifications and implementations.

Specifications:

Implementations:

The concept of CBI session manager (cbism) is decoupled from cbid, so as to make cbid free from I/O overhead.

Build context

CBI defines the following values for context.kind:

Plugin implementations SHOULD implement ConfigMap, Git, and HTTP, but none of them is mandatory. Also, implementations MAY accept non-standard context.kind values.

For ease of implementation of these context providers, CBI provides cbipluginhelper image that contains CLI utilities for accessing these remote contexts. Plugin implementations may inject the cbipluginhelper image with an emptyDir volume into the initContainers of the job pods, so as to support contexts that are not natively supported by the backends.

e.g. CBI plugin for Docker (cbi-docker) supports Rclone context using cbipluginhelper, while Docker itself does not support Rclone.

BuildkitSession (Planned)

If BuildkitSession is specified as context.kind, the pod ID of a CBI session manager, TCP port number, and the session ID would be set to the status fields of the BuildJob object.

The client is expected to send the context to the specified session manager pod using BuildKit session gRPC (via the HTTP/1.1 gate). To connect to the pod, the client may use kubectl port-forward or kubectl exec ... socat.

Future version would also provide Ingress for exposing the CBI session manager in more efficient ways.

Contribute to CBI

Testing

$ ./hack/build/build-push-apply.sh your-registry.example.com:5000/cbi test20180501

This command performs:

Local testing with DinD

You may use hack/dind/up.sh for setting up a local Kubernetes cluster and a local registry using Docker-in-Docker.

$ ./hack/dind/up.sh
$ DOCKER_HOST=localhost:62375 ./hack/build/build-push-apply.sh cbi-registry:5000/cbi test20180501
$ ./hack/dind/down.sh

The Kubernetes cluster and the "bootstrap" Docker listening at localhost:62375 can connect to cbi-registry:5000 without auth.

FAQs

Q: Does CBI standardize the Dockerfile specification?

A: No, the Dockerfile specification has been maintained by Docker, Inc.

CBI itself is neutral to any image building instruction language (e.g. Dockerfile).

However, most backend implementations would accept Dockerfile.

Q: Does CBI replace BuildKit?

A: No, CBI just provides an abstract interface for several backends such as BuildKit.

Q: Is CBI a part of Kubernetes, a Kubernetes incubator, or a CNCF project?

A: Currently no, unlike CRI/CNI/CSI.

But it'd be good to donate CBI to such a vendor-neutral organization if CBI becomes popular.