Cleanup docker images after building e2e binary

e2eバイナリの作成を繰り返していると、ルートファイルシステムの領域を侵食してしまう。 e2eバイナリ作成用のdocker imageからバイナリ取得後、そのdocker imageは不要なはずなのでクリーンアップ処理を追加する。

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        78G   51G   27G  66% /
...
$ sudo docker images
REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google-containers/debian-iptables-amd64   v10                 196b441f5192        12 months ago       45.7MB
k8s.gcr.io/debian-iptables-amd64                 v10                 196b441f5192        12 months ago       45.7MB
$ git branch
  issue/74326
  master
* v1.13.1
$ sudo /usr/local/go/bin/go  run hack/e2e.go -- --build
2019/02/26 17:32:06 e2e.go:153: The kubetest binary is older than 24h0m0s.
2019/02/26 17:32:06 e2e.go:158: Updating kubetest binary...
2019/02/26 17:32:44 e2e.go:79: Calling kubetest --verbose-commands=true --build...
2019/02/26 17:32:44 process.go:153: Running: make -C /home/ubuntu/go/src/k8s.io/kubernetes quick-release
make: Entering directory '/home/ubuntu/go/src/k8s.io/kubernetes'
+++ [0226 17:32:44] Verifying Prerequisites....
+++ [0226 17:32:48] Building Docker image kube-build:build-127ba8f02e-5-v1.11.2-1
+++ [0226 17:34:47] Creating data container kube-build-data-127ba8f02e-5-v1.11.2-1
+++ [0226 17:34:50] Syncing sources to container
+++ [0226 17:35:22] Running build command...
+++ [0226 17:35:33] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0226 17:35:39] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0226 17:35:43] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0226 17:35:48] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
2019/02/26 17:35:53 Code for OpenAPI definitions generated
+++ [0226 17:35:53] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0226 17:35:54] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager
    cmd/kubelet
    cmd/kubeadm
    cmd/hyperkube
    cmd/kube-scheduler
    vendor/k8s.io/apiextensions-apiserver
    cluster/gce/gci/mounter
+++ [0226 17:38:59] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kubeadm
    cmd/kubelet
+++ [0226 17:39:20] Building go targets for linux/amd64:
    cmd/kubectl
+++ [0226 17:39:30] Building go targets for linux/amd64:
    cmd/gendocs
    cmd/genkubedocs
    cmd/genman
    cmd/genyaml
    cmd/genswaggertypedocs
    cmd/linkcheck
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e/e2e.test
+++ [0226 17:42:01] Building go targets for linux/amd64:
    cmd/kubemark
    vendor/github.com/onsi/ginkgo/ginkgo
    test/e2e_node/e2e_node.test
+++ [0226 17:42:44] Syncing out of container
+++ [0226 17:43:23] Building tarball: manifests
+++ [0226 17:43:23] Building tarball: src
+++ [0226 17:43:23] Starting tarball: client linux-amd64
+++ [0226 17:43:23] Waiting on tarballs
+++ [0226 17:43:59] Building tarball: node linux-amd64
+++ [0226 17:43:59] Building images: linux-amd64
+++ [0226 17:43:59] Starting docker build for image: cloud-controller-manager-amd64
+++ [0226 17:43:59] Starting docker build for image: kube-apiserver-amd64
+++ [0226 17:43:59] Starting docker build for image: kube-controller-manager-amd64
+++ [0226 17:43:59] Starting docker build for image: kube-scheduler-amd64
+++ [0226 17:43:59] Starting docker build for image: kube-proxy-amd64
+++ [0226 17:43:59] Building hyperkube image for arch: amd64
+++ [0226 17:43:59] Building conformance image for arch: amd64
+++ [0226 17:44:33] Deleting docker image k8s.gcr.io/kube-proxy:v1.13.1-dirty
+++ [0226 17:44:33] Deleting docker image k8s.gcr.io/kube-scheduler:v1.13.1-dirty
+++ [0226 17:44:33] Deleting docker image k8s.gcr.io/kube-apiserver:v1.13.1-dirty
+++ [0226 17:44:44] Deleting docker image k8s.gcr.io/cloud-controller-manager:v1.13.1-dirty
+++ [0226 17:44:45] Deleting docker image k8s.gcr.io/kube-controller-manager:v1.13.1-dirty
+++ [0226 17:45:03] Deleting hyperkube image k8s.gcr.io/hyperkube-amd64:v1.13.1-dirty
+++ [0226 17:45:11] Deleting conformance image k8s.gcr.io/conformance-amd64:v1.13.1-dirty
+++ [0226 17:45:11] Docker builds done
+++ [0226 17:45:11] Building tarball: server linux-amd64
+++ [0226 17:46:10] Building tarball: final
+++ [0226 17:46:10] Building tarball: test
make: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes'
2019/02/26 17:46:54 process.go:155: Step 'make -C /home/ubuntu/go/src/k8s.io/kubernetes quick-release' finished in 14m9.59444311s
2019/02/26 17:46:54 e2e.go:83: Done
$

実行後の状態

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        78G   59G   19G  76% /
...
$ sudo docker images
REPOSITORY                                       TAG                            IMAGE ID            CREATED             SIZE
kube-build                                       build-127ba8f02e-5-v1.11.2-1   d8e713ce1cb4        About an hour ago   2.3GB
k8s.gcr.io/kube-cross                            v1.11.2-1                      902cf4721f5c        3 months ago        1.7GB
k8s.gcr.io/debian-hyperkube-base-amd64           0.12.0                         a6aee83339eb        4 months ago        387MB
k8s.gcr.io/debian-iptables-amd64                 v11.0                          48319fdf4d25        4 months ago        45.4MB
k8s.gcr.io/debian-base-amd64                     0.4.0                          8021d54711e6        4 months ago        42.3MB
gcr.io/google-containers/debian-iptables-amd64   v10                            196b441f5192        12 months ago       45.7MB
k8s.gcr.io/debian-iptables-amd64                 v10                            196b441f5192        12 months ago       45.7MB

kube-build                                       build-127ba8f02e-5-v1.11.2-1   d8e713ce1cb4        About an hour ago   2.3GB
k8s.gcr.io/kube-cross                            v1.11.2-1                      902cf4721f5c        3 months ago        1.7GB
k8s.gcr.io/debian-hyperkube-base-amd64           0.12.0                         a6aee83339eb        4 months ago        387MB
k8s.gcr.io/debian-iptables-amd64                 v11.0                          48319fdf4d25        4 months ago        45.4MB
k8s.gcr.io/debian-base-amd64                     0.4.0                          8021d54711e6        4 months ago        42.3MB

が増えており、8GB消費していることがわかる。一方、ログを見ると docker images のクリーンアップ処理はあるようなので、ここに上記のクリーンアップ漏れしているイメージも指定すれば良さそう。

一番大きな kube-build イメージの削除を目指す。 kube-build の作成処理は以下 k/k

438 # Build a docker image from a Dockerfile.
439 # $1 is the name of the image to build
440 # $2 is the location of the "context" directory, with the Dockerfile at the root.
441 # $3 is the value to set the --pull flag for docker build; true by default
442 function kube::build::docker_build() {
443   local -r image=$1
444   local -r context_dir=$2
445   local -r pull="${3:-true}"
446   local -ra build_cmd=("${DOCKER[@]}" build -t "${image}" "--pull=${pull}" "${context_dir}")
447
448   kube::log::status "Building Docker image ${image}"
449   local docker_output
450   docker_output=$("${build_cmd[@]}" 2>&1) || {
451     cat <<EOF >&2
452 +++ Docker build command failed for ${image}
453
454 ${docker_output}
455
456 To retry manually, run:
457
458 ${build_cmd[*]}
459
460 EOF
461     return 1
462   }
463 }

で、下記で上記関数を呼出

426   kube::build::docker_build "${KUBE_BUILD_IMAGE}" "${LOCAL_OUTPUT_BUILD_CONTEXT}" 'false'
427
428   # Clean up old versions of everything
429   kube::build::docker_delete_old_containers "${KUBE_BUILD_CONTAINER_NAME_BASE}" "${KUBE_BUILD_CONTAINER_NAME}"
430   kube::build::docker_delete_old_containers "${KUBE_RSYNC_CONTAINER_NAME_BASE}" "${KUBE_RSYNC_CONTAINER_NAME}"
431   kube::build::docker_delete_old_containers "${KUBE_DATA_CONTAINER_NAME_BASE}" "${KUBE_DATA_CONTAINER_NAME}"
432   kube::build::docker_delete_old_images "${KUBE_BUILD_IMAGE_REPO}" "${KUBE_BUILD_IMAGE_TAG_BASE}" "${KUBE_BUILD_IMAGE_TAG}"
433
434   kube::build::ensure_data_container
435   kube::build::sync_to_container
436 }

k/kubernetes/pull/74624 で対応。こちらのチケットはクローズする。

作業メモ

kube-build の使われ方を把握する

+++ [0226 17:32:48] Building Docker image kube-build:build-127ba8f02e-5-v1.11.2-1
+++ [0226 17:34:47] Creating data container kube-build-data-127ba8f02e-5-v1.11.2-1
+++ [0226 17:34:50] Syncing sources to container
+++ [0226 17:35:22] Running build command...
+++ [0226 17:35:33] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/deepcopy-gen
+++ [0226 17:35:39] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/defaulter-gen
+++ [0226 17:35:43] Building go targets for linux/amd64:
    ./vendor/k8s.io/code-generator/cmd/conversion-gen
+++ [0226 17:35:48] Building go targets for linux/amd64:
    ./vendor/k8s.io/kube-openapi/cmd/openapi-gen
2019/02/26 17:35:53 Code for OpenAPI definitions generated
+++ [0226 17:35:53] Building go targets for linux/amd64:
    ./vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0226 17:35:54] Building go targets for linux/amd64:
    cmd/kube-proxy
    cmd/kube-apiserver
    cmd/kube-controller-manager
    cmd/cloud-controller-manager

関連ディレクトリは k8s.io/kubernetes/build

release.sh
    kube::build::build_image() { - common.sh
        kube::build::docker_build
            kube::log::status "Building Docker image ${image}"
        kube::build::ensure_data_container
        kube::build::sync_to_container
            kube::log::status "Syncing sources to container"
    kube::build::copy_output
        kube::log::status "Syncing out of container"
    kube::release::package_tarballs
        kube::release::package_server_tarballs
            kube::log::status "Building tarball: server $platform_tag"
    ここにクリーンアップを追加すれば良さそう

他の削除処理
293   kube::log::status "Deleting kube-build image ${KUBE_BUILD_IMAGE}"
294   "${DOCKER[@]}" rmi "${KUBE_BUILD_IMAGE}" &>/dev/null || true

その前に使っているDockerコンテナを削除する必要があり
$ sudo docker ps -a
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS                   PORTS               NAMES
a1081de8e99d        kube-build:build-127ba8f02e-5-v1.11.2-1   "chown -R 0:0 /go/sr・   3 hours ago         Exited (0) 3 hours ago                       kube-build-data-127ba8f02e-5-v1.11.2-1

oomichi / try-kubernetes

Cleanup docker images after building e2e binary #75