kubepkg tests fail

[hoegaarden]

Running go test ./... fails for the package k8s.io/release/pkg/kubepkg over here.

time="2020-01-08T14:22:29Z" level=info msg="Setting version for  package..."
time="2020-01-08T14:22:29Z" level=info msg="Using Kubernetes version for  package"
time="2020-01-08T14:22:29Z" level=info msg="Setting version for  package..."
time="2020-01-08T14:22:29Z" level=info msg="Using Kubernetes version for  package"
time="2020-01-08T14:22:29Z" level=info msg="Setting version for kubernetes-cni package..."
time="2020-01-08T14:22:29Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:29Z" level=info msg="Setting CNI version to 0.8.3"
time="2020-01-08T14:22:29Z" level=info msg="Setting version for cri-tools package..."
time="2020-01-08T14:22:29Z" level=info msg="Getting CRI version..."
--- FAIL: TestGetPackageVersionSuccess (0.66s)
    kubepkg_test.go:68: did not expect an error: GET https://api.github.com/repos/kubernetes-sigs/cri-tools/releases: 403 API rate limit exceeded for <redacted>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 22m17s]
time="2020-01-08T14:22:30Z" level=info msg="Using Kubernetes version (1.17.0) for  package"
time="2020-01-08T14:22:31Z" level=info msg="Retrieved Kubernetes version: 1.14.10"
time="2020-01-08T14:22:31Z" level=info msg="Retrieved Kubernetes version: 1.14.11-beta.1"
time="2020-01-08T14:22:31Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:31Z" level=info msg="Kubernetes version earlier than 1.17 must use CNI version <= 0.7.5"
time="2020-01-08T14:22:31Z" level=info msg="Setting CNI version to 0.7.5"
time="2020-01-08T14:22:31Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:31Z" level=info msg="Setting CNI version to 0.8.3"
time="2020-01-08T14:22:31Z" level=info msg="Getting CNI version..."
time="2020-01-08T14:22:31Z" level=info msg="Setting CNI version to 0.7.5"
time="2020-01-08T14:22:31Z" level=info msg="Getting CRI version..."
--- FAIL: TestGetCRIToolsVersionSuccess (0.14s)
    kubepkg_test.go:252: did not expect an error: GET https://api.github.com/repos/kubernetes-sigs/cri-tools/releases: 403 API rate limit exceeded for <redacted>. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) [rate reset in 22m16s]
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of kubelet check"
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of release check"
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of testing check"
time="2020-01-08T14:22:31Z" level=info msg="Adding 'fakearch' (type: string) to not supported"
time="2020-01-08T14:22:31Z" level=info msg="Breaking out of amd64 check"
time="2020-01-08T14:22:31Z" level=info msg="The following options are not supported: [fakearch]"
FAIL
FAIL    k8s.io/release/pkg/kubepkg      2.197s

This indicates IMHO that we should take more care about our tests and split them into different types (static, unit, integration, e2e).

[hoegaarden]

/kind failing-test /priority important-soon /area release-eng

[justaugustus]

@hoegaarden -- Here's what I got, FWIW:

$ go test ./...
?       k8s.io/release/cmd/blocking-testgrid-tests  [no test files]
?       k8s.io/release/cmd/krel [no test files]
?       k8s.io/release/cmd/krel/cmd [no test files]
?       k8s.io/release/cmd/kubepkg  [no test files]
?       k8s.io/release/cmd/kubepkg/cmd  [no test files]
?       k8s.io/release/cmd/release-notes    [no test files]
ok      k8s.io/release/pkg/command  0.020s
ok      k8s.io/release/pkg/git  0.990s
ok      k8s.io/release/pkg/kubepkg  1.754s
ok      k8s.io/release/pkg/notes    0.356s
ok      k8s.io/release/pkg/notes/internal   0.004s
?       k8s.io/release/pkg/notes/notesfakes [no test files]
ok      k8s.io/release/pkg/release  1.011s
ok      k8s.io/release/pkg/util 1.003s

How would you propose organizing the tests?

[hoegaarden]

How would you propose organizing the tests?

Not sure :) I have seen different approaches to separate different test types. Wanted to bring them up and compare them and start a discussion. Stay tuned.

[justaugustus]

SGTM! I've mentioned on a few calls that testing methodologies is not a strong suit of mine, so happy to have you to steer the convo.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

[saschagrunert]

/remove-lifecycle rotten

[justaugustus]

/assign @saschagrunert /milestone v1.19

[saschagrunert]

I think our counterfeiter mocking approach works out pretty well to isolate the tests. We're still in a transition phase and some tests are connecting to the internet, which might cause API rate limit issues. I'll check what I can do for the kubepkg tests especially.

[justaugustus]

Thanks @saschagrunert!