conformance: Suite panics on API server warning logs if no or nil logger set

sunjayBhatia commented 1 year ago

What happened:

In Contour's implementation of the conformance test suite (and the case in the Gateway API provided suite), we do not call the controller-runtime log.SetLogger() helper to set the client log sink. controller-runtime will attempt to log API server warnings and when it encounters a nil/unset log sink will panic.

[controller-runtime] log.SetLogger(...) was never called, logs will not be displayed:
goroutine 151 [running]:
runtime/debug.Stack()
    /opt/hostedtoolcache/go/1.20.4/x64/src/runtime/debug/stack.go:24 +0x65
sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/log.go:59 +0xbd
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc00044c0c0, 0x8c00000000000000?)
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/deleg.go:111 +0x3d
github.com/go-logr/logr.Logger.Enabled(...)
    /home/runner/go/pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:261
github.com/go-logr/logr.Logger.Info({{0x1a03688?, 0xc00044c0c0?}, 0xc00045fb80?}, {0xc00045fb80, 0x35}, {0x0, 0x0, 0x0})
    /home/runner/go/pkg/mod/github.com/go-logr/logr@v1.2.4/logr.go:274 +0x78
sigs.k8s.io/controller-runtime/pkg/log.(*KubeAPIWarningLogger).HandleWarningHeader(0xc000485050?, 0x1?, {0x17887f5?, 0x7?}, {0xc00045fb80?, 0x35?})
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/log/warning_handler.go:65 +0x170
k8s.io/client-go/rest.handleWarnings(0xc00002b2f0?, {0x19edae0?, 0xc000485050?})
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/warnings.go:144 +0xfc
k8s.io/client-go/rest.(*Request).transformResponse(0xc00002d8c0, 0xc0004c42d0, 0xc0003d0900)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/request.go:1[155](https://github.com/projectcontour/contour/actions/runs/5149469808/jobs/9329371054#step:7:156) +0x5cf
k8s.io/client-go/rest.(*Request).Do.func1(0xc00015d780?, 0x0?)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/request.go:1040 +0x45
k8s.io/client-go/rest.(*Request).request.func3.1(...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/request.go:1015
k8s.io/client-go/rest.(*Request).request.func3(0xc0004c42d0, 0xc00066d8a0, {0x1a01d00?, 0xc00015d780?}, 0x0?, 0x0?, 0x171bc20?, {0x0?, 0x0?}, 0x18a9948)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/request.go:1022 +0xde
k8s.io/client-go/rest.(*Request).request(0xc00002d8c0, {0x1a01910, 0xc00002b1d0}, 0x1?)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/request.go:1024 +0x4e5
k8s.io/client-go/rest.(*Request).Do(0xc00002d8c0, {0x1a01910, 0xc00002b1d0})
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/rest/request.go:1039 +0xc9
sigs.k8s.io/controller-runtime/pkg/client.(*unstructuredClient).Create(0xc000498b58, {0x1a01910, 0xc00002b1d0}, {0x1a121d8?, 0xc00015c9e0?}, {0x0, 0x0, 0x1551f40?})
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/client/unstructured_client.go:57 +0x486
sigs.k8s.io/controller-runtime/pkg/client.(*client).Create(0xc000498b40, {0x1a01910, 0xc00002b1d0}, {0x1a121d8?, 0xc00015c9e0}, {0x0, 0x0, 0x0})
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/client/client.go:278 +0x136
sigs.k8s.io/gateway-api/conformance/utils/kubernetes.Applier.MustApplyWithCleanup({0x0, {0x7ffed4b64ce5, 0x7}, {0xc0002f70b0, 0x24}, {0x1a20c40}}, 0xc00013a9c0, {0x1a07740?, 0xc000498b40}, {0xdf8475800, ...}, ...)
    /home/runner/go/pkg/mod/sigs.k8s.io/gateway-api@v0.7.1/conformance/utils/kubernetes/apply.go:178 +0x75c
sigs.k8s.io/gateway-api/conformance/utils/suite.(*ConformanceTest).Run(0xc0004a8720, 0xc00013a9c0, 0xc00042c640)
    /home/runner/go/pkg/mod/sigs.k8s.io/gateway-api@v0.7.1/conformance/utils/suite/suite.go:226 +0x3f4
sigs.k8s.io/gateway-api/conformance/utils/suite.(*ConformanceTestSuite).Run.func1(0xc0002ac0f0?)
    /home/runner/go/pkg/mod/sigs.k8s.io/gateway-api@v0.7.1/conformance/utils/suite/suite.go:187 +0x27
testing.tRunner(0xc00013a9c0, 0xc000013578)
    /opt/hostedtoolcache/go/1.20.4/x64/src/testing/testing.go:[157](https://github.com/projectcontour/contour/actions/runs/5149469808/jobs/9329371054#step:7:158)6 +0x10b
created by testing.(*T).Run
    /opt/hostedtoolcache/go/1.20.4/x64/src/testing/testing.go:[162](https://github.com/projectcontour/contour/actions/runs/5149469808/jobs/9329371054#step:7:163)9 +0x3ea
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc0 pc=0x1440b24]

goroutine 2728 [running]:
k8s.io/client-go/kubernetes.(*Clientset).CoreV1(...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.27.2/kubernetes/clientset.go:309
sigs.k8s.io/gateway-api/conformance/utils/kubernetes.DumpEchoLogs({0x[179](https://github.com/projectcontour/contour/actions/runs/5149469808/jobs/9329371054#step:7:180)d60a, 0x19}, {0x179208b, 0x10}, {0x1a07740, 0xc000498b40}, 0x0)
    /home/runner/go/pkg/mod/sigs.k8s.io/gateway-api@v0.7.1/conformance/utils/kubernetes/logs.go:50 +0x3c4
sigs.k8s.io/gateway-api/conformance/utils/http.ExpectMirroredRequest.func1()
    /home/runner/go/pkg/mod/sigs.k8s.io/gateway-api@v0.7.1/conformance/utils/http/mirror.go:43 +0x1eb
github.com/stretchr/testify/assert.Eventually.func1()
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.3/assert/assertions.go:[185](https://github.com/projectcontour/contour/actions/runs/5149469808/jobs/9329371054#step:7:186)2 +0x29
created by github.com/stretchr/testify/assert.Eventually
    /home/runner/go/pkg/mod/github.com/stretchr/testify@v1.8.3/assert/assertions.go:1852 +0x22a
FAIL    github.com/projectcontour/contour/test/conformance/gatewayapi   876.354s
FAIL

What you expected to happen:

controller-runtime should not panic, that is getting fixed in a future release: https://github.com/kubernetes-sigs/controller-runtime/issues/2352
The Gateway API conformance suite could either set a log sink to surface API server warnings or suppress the warnings entirely

How to reproduce it (as minimally and precisely as possible):

Run the conformance suite until you have an API server return warnings to the controller-runtime client

Anything else we need to know?:

Listed as a bug report here, mostly as informational for other implementers and maintainers. The solution may just be to wait until controller-runtime fixes the bug.

k8s-triage-robot commented 2 months ago

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

Confirm that this issue is still relevant with /triage accepted (org members only)
Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

robscott commented 2 months ago

/help /triage accepted

k8s-ci-robot commented 2 months ago

@robscott: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

Why are we solving this issue?
To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
Does this issue have zero to low barrier of entry?
How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes-sigs/gateway-api/issues/2090): >/help >/triage accepted Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

robscott commented 2 months ago

Actually, it looks like this was fixed https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.16.0 and we're already on v0.18, just looks like we forgot about this issue. Closing this out.

kubernetes-sigs / gateway-api

conformance: Suite panics on API server warning logs if no or nil logger set #2090

Guidelines