search

oomichi / try-kubernetes

11 stars 5 forks source link

Update e2e way #100

Closed oomichi closed 4 years ago

oomichi commented 4 years ago

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md の手順をローカル環境で試し、このリポジトリの手順も更新する。 必要に応じて PR を提案する。

oomichi commented 4 years ago

未だに以下のエラーが出る

$ go get -u k8s.io/test-infra/kubetest
# k8s.io/client-go/transport
go/src/k8s.io/client-go/transport/round_trippers.go:70:11: cannot convert klog.V(9) (type klog.Verbose) to type bool
go/src/k8s.io/client-go/transport/round_trippers.go:72:11: cannot convert klog.V(8) (type klog.Verbose) to type bool
go/src/k8s.io/client-go/transport/round_trippers.go:74:11: cannot convert klog.V(7) (type klog.Verbose) to type bool
go/src/k8s.io/client-go/transport/round_trippers.go:76:11: cannot convert klog.V(6) (type klog.Verbose) to type bool
oomichi commented 4 years ago

最新の手順として

$ GO111MODULE=on go get k8s.io/test-infra/kubetest

で kubetest をビルドするのが正しいらしい。 しかし、以下のようにビルドでエラーになる。

$ GO111MODULE=on go get k8s.io/test-infra/kubetest
...
go: finding golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: finding k8s.io/utils v0.0.0-20190506122338-8fab8cb257d5
go: finding golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45
# k8s.io/client-go/rest
go/pkg/mod/k8s.io/client-go@v11.0.1-0.20190805182717-6502b5e7b1b5+incompatible/rest/request.go:598:31: not enough arguments in call to watch.NewStreamWatcher
        have (*versioned.Decoder)
        want (watch.Decoder, watch.Reporter)

最新の client-go の内容を見ると、以下のように want で書かれている内容になっている。

 608         return watch.NewStreamWatcher(
 609                 restclientwatch.NewDecoder(wrapperDecoder, embeddedDecoder),
 610                 // use 500 to indicate that the cause of the error is unknown - other error codes
 611                 // are more specific to HTTP interactions, and set a reason
 612                 errors.NewClientErrorReporter(http.StatusInternalServerError, r.verb, "ClientWatchDecoding"),
 613         ), nil

2019年2月に https://github.com/kubernetes/client-go/commit/cd12199def5832afb81d78426c3b2758ee70110f で変更になっている。 go/pkg/mod/k8s.io/client-go@v11.0.1-0.20190805182717-6502b5e7b1b5 から近い v11.0.0 の実装からすると

598         return watch.NewStreamWatcher(restclientwatch.NewDecoder(wrapperDecoder, embeddedDecoder)), nil
oomichi commented 4 years ago

watch は k8s.io/apimachinery/pkg/watch

 61 // NewStreamWatcher creates a StreamWatcher from the given decoder.
 62 func NewStreamWatcher(d Decoder, r Reporter) *StreamWatcher {
 63         sw := &StreamWatcher{
 64                 source:   d,
 65                 reporter: r,
 66                 // It's easy for a consumer to add buffering via an extra
 67                 // goroutine/channel, but impossible for them to remove it,
 68                 // so nonbuffered is better.
 69                 result: make(chan Event),
 70         }
 71         go sw.receive()
 72         return sw
 73 }

以下の Commit で変わった。

commit 6bfc4729640e1b788e991515a8ea4e16504c3a5d
Author: Clayton Coleman <ccoleman@redhat.com>
Date:   Mon Feb 11 17:34:20 2019 -0500

    Report a watch error instead of eating it when we can't decode

    Clients are required to handle watch events of type ERROR, so instead
    of eating the decoding error we should pass it on to the client. Use
    NewGenericServerError with isUnexpectedResponse to indicate that we
    didn't get the bytes from the server we were expecting. For watch, the
    415 error code is roughly correct and we will return an error to the
    client that makes debugging a failure in either server watch or client
    machinery much easier.

    We do not alter the behavior when it appears the response is an EOF
    or other disconnection.

    Kubernetes-commit: 89620d5667adec6c132b2713b79efb1dd2391723

client-go ビルドのタイミングでなぜ上記の前の apimachinery が取れなかったのか・・

oomichi commented 4 years ago

download のログを見ると client-go は v11.0.1 Tag のをダウンロードしているが、apimachinery は v0.0.0 のものをダウンロードしようとしている。

go: downloading k8s.io/client-go v11.0.1-0.20190805182717-6502b5e7b1b5+incompatible
go: downloading k8s.io/apimachinery v0.0.0-20190817020851-f2f3a405f61d

apimachinery には Tag がついていなくて、常に最新のものをダウンロードしている？ client-go には既存の versioning scheme (x.y.z) に対応した Tag がついているが、apimachinery には kubernetes-1.16.0 などの Tag のみで x.y.z の Tag がついていない。 （kubernetes-1.16.0 などの Tag も client-go にはついている） これにより、apimachinery では常に最新のコードがダウンロードされている？ → ただしい。k/kubernetes/issues/76304 が登録されている。

oomichi commented 4 years ago

ソースコードからの Build は何もしなくても成功する。

$ go get k8s.io/test-infra
$ cd go/src/k8s.io/test-infra/kubetest
$ go build
...
go: downloading k8s.io/apimachinery v0.0.0-20190817020851-f2f3a405f61d
go: extracting k8s.io/client-go v0.0.0-20190918200256-06eb1244587a
...

client-go、apimachinery ともに最新 master がダウンロードされている模様

oomichi commented 4 years ago

コミュニティの手順に合わせようとすると、意外に下記のように面倒

$ go get k8s.io/kubernetes
$ sudo apt-get install -y docker.io gcc make
$ cd go/src/k8s.io/test-infra/kubetest
$ sudo ./kubetest --build

更新しなくてよさげ

oomichi commented 4 years ago

しかも build でエラーになった

$ sudo ./kubetest --build
...
    test/e2e/e2e.test
    cluster/images/conformance/go-runner
# k8s.io/kubernetes/cmd/genman
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: final link failed: memory exhausted
collect2: error: ld returned 1 exit status

!!! [0927 03:13:54] Call tree:
!!! [0927 03:13:54]  1: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:714 kube::golang::build_some_binaries(...)
!!! [0927 03:13:54]  2: /go/src/k8s.io/kubernetes/hack/lib/golang.sh:853 kube::golang::build_binaries_for_platform(...)
!!! [0927 03:13:54]  3: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0927 03:13:54] Call tree:
!!! [0927 03:13:54]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
!!! [0927 03:13:54] Call tree:
!!! [0927 03:13:54]  1: hack/make-rules/build.sh:27 kube::golang::build_binaries(...)
make[1]: *** [Makefile:93: all] Error 1
make: *** [Makefile:522: cross] Error 1
!!! [0927 03:13:54] Call tree:
!!! [0927 03:13:55]  1: build/../build/common.sh:521 kube::build::run_build_command_ex(...)
!!! [0927 03:13:55]  2: build/release.sh:36 kube::build::run_build_command(...)
Makefile:465: recipe for target 'quick-release' failed
make: *** [quick-release] Error 1
make: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes'
2019/09/27 03:13:55 process.go:155: Step 'make -C /home/ubuntu/go/src/k8s.io/kubernetes quick-release' finished in 11m49.571778752s
2019/09/27 03:13:55 main.go:319: Something went wrong: failed to acquire k8s binaries: error during make -C /home/ubuntu/go/src/k8s.io/kubernetes quick-release: exit status 2

memory exhausted ってメモリ不足！？ 8GB割当てているのに？

oomichi commented 4 years ago

これまでこのリポジトリに書いていた手順ではビルドできなくなっている。 たぶん、この手順は陳腐化していて誰もメンテナンスしなくなっている。 -> やっぱり、更新しないと駄目っぽい。

$ sudo /usr/local/go/bin/go  run hack/e2e.go -- --build
..
go: finding k8s.io/utils v0.0.0-20190920012459-5008bf6f8cd6
build k8s.io/test-infra/kubetest: cannot load github.com/Azure/go-autorest/autorest: ambiguous import: found github.com/Azure/go-autorest/autorest in multiple modules:
        github.com/Azure/go-autorest v11.3.1+incompatible (/home/ubuntu/go/pkg/mod/github.com/!azure/go-autorest@v11.3.1+incompatible/autorest)
        github.com/Azure/go-autorest/autorest v0.9.0 (/home/ubuntu/go/pkg/mod/github.com/!azure/go-autorest/autorest@v0.9.0)
2019/09/27 17:39:05 e2e.go:77: err: go get -u k8s.io/test-infra/kubetest: exit status 1
exit status 1
oomichi commented 4 years ago

16GB の VM 上で実行した。 -> k8s.io/kubernetes ディレクトリで kubetest を実行すべきだった模様

    test/e2e_node/e2e_node.test
+++ [0927 18:42:41] Syncing out of container
+++ [0927 18:42:53] Building tarball: src
+++ [0927 18:42:53] Building tarball: manifests
+++ [0927 18:42:53] Starting tarball: client linux-amd64
+++ [0927 18:42:53] Waiting on tarballs
+++ [0927 18:43:00] Building tarball: node linux-amd64
+++ [0927 18:43:00] Building images: linux-amd64
+++ [0927 18:43:01] Starting docker build for image: kube-apiserver-amd64
+++ [0927 18:43:01] Starting docker build for image: kube-controller-manager-amd64
+++ [0927 18:43:01] Starting docker build for image: kube-scheduler-amd64
+++ [0927 18:43:01] Starting docker build for image: kube-proxy-amd64
+++ [0927 18:43:01] Building conformance image for arch: amd64
+++ [0927 18:43:01] Building hyperkube image for arch: amd64
+++ [0927 18:43:16] Deleting docker image k8s.gcr.io/kube-proxy-amd64:v1.17.0-alpha.0.1880_cd95e57c71a194
+++ [0927 18:43:17] Deleting docker image k8s.gcr.io/kube-scheduler-amd64:v1.17.0-alpha.0.1880_cd95e57c71a194
+++ [0927 18:43:20] Deleting docker image k8s.gcr.io/kube-controller-manager-amd64:v1.17.0-alpha.0.1880_cd95e57c71a194
+++ [0927 18:43:21] Deleting docker image k8s.gcr.io/kube-apiserver-amd64:v1.17.0-alpha.0.1880_cd95e57c71a194
+++ [0927 18:43:39] Deleting hyperkube image k8s.gcr.io/hyperkube-amd64:v1.17.0-alpha.0.1880_cd95e57c71a194
+++ [0927 18:43:45] Deleting conformance image k8s.gcr.io/conformance-amd64:v1.17.0-alpha.0.1880_cd95e57c71a194
+++ [0927 18:43:45] Docker builds done
+++ [0927 18:43:45] Building tarball: server linux-amd64
+++ [0927 18:45:15] Building tarball: final
+++ [0927 18:45:15] Waiting on test tarballs
+++ [0927 18:45:15] Starting tarball: test linux-amd64
+++ [0927 18:46:15] Building tarball: test portable
make: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes'
2019/09/27 18:46:15 process.go:155: Step 'make -C /home/ubuntu/go/src/k8s.io/kubernetes quick-release' finished in 18m39.159745513s
2019/09/27 18:46:15 main.go:319: Something went wrong: called from invalid working directory: must run from kubernetes directory root. current: /home/ubuntu/go/src/k8s.io/test-infra/kubetest
$ echo $?
1
$

下記の手順で実施。 -> 成功

~/go/src/k8s.io/kubernetes$ sudo ../test-infra/kubetest/kubetest --build
..
+++ [0927 18:56:49] Docker builds done
+++ [0927 18:56:49] Building tarball: server linux-amd64
+++ [0927 18:58:19] Building tarball: final
+++ [0927 18:58:19] Waiting on test tarballs
+++ [0927 18:58:19] Starting tarball: test linux-amd64
+++ [0927 18:59:18] Building tarball: test portable
make: Leaving directory '/home/ubuntu/go/src/k8s.io/kubernetes'
2019/09/27 18:59:18 process.go:155: Step 'make -C /home/ubuntu/go/src/k8s.io/kubernetes quick-release' finished in 9m58.888311574s
$
oomichi commented 4 years ago

Conformance Test の実行方法が少し変わっている。 https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md

kubetest コマンドを利用して実行するようになっている。

$ mkdir ${HOME}/go/bin
$ cp ${HOME}/go/src/k8s.io/test-infra/kubetest/kubetest ${HOME}/go/bin
$ export KUBECONFIG=${HOME}/admin.conf
$ export KUBERNETES_CONFORMANCE_TEST=y
$ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" --check-version-skew=false