search

spinkube / spin-operator

Spin Operator is a Kubernetes operator that empowers platform engineers to deploy Spin applications as custom resources to their Kubernetes clusters
https://www.spinkube.dev/docs/overview/
Other
185 stars 23 forks source link

Flaky integration test #17

Closed calebschoepp closed 8 months ago

calebschoepp commented 9 months ago

I'm experiencing the integration tests flaking occasionally because of a race condition. I've been unable to diagnose the issue so far but I think it has something to do with this code.

test -s /Users/caleb/fermyon/spin-operator/bin/controller-gen && /Users/caleb/fermyon/spin-operator/bin/controller-gen --version | grep -q v0.13.0 || \
    GOBIN=/Users/caleb/fermyon/spin-operator/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.13.0
/Users/caleb/fermyon/spin-operator/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./api/..." paths="./cmd/..." paths="./internal/..." paths="./pkg/..." output:crd:artifacts:config=config/crd/bases
/Users/caleb/fermyon/spin-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./api/..." paths="./cmd/..." paths="./internal/..." paths="./pkg/..."
go fmt ./...
go vet ./...
test -s /Users/caleb/fermyon/spin-operator/bin/setup-envtest || GOBIN=/Users/caleb/fermyon/spin-operator/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
/Users/caleb/fermyon/spin-operator/bin/setup-envtest use 1.28.3 --bin-dir /Users/caleb/fermyon/spin-operator/bin
Version: 1.28.3
OS/Arch: darwin/arm64
Path: /Users/caleb/fermyon/spin-operator/bin/k8s/1.28.3-darwin-arm64
KUBEBUILDER_ASSETS="/Users/caleb/fermyon/spin-operator/bin/k8s/1.28.3-darwin-arm64" go test ./... -coverprofile cover.out
?       github.com/fermyon/spin-operator/api/v1 [no test files]
?       github.com/fermyon/spin-operator/cmd    [no test files]
?       github.com/fermyon/spin-operator/internal/constants [no test files]
?       github.com/fermyon/spin-operator/internal/logging   [no test files]
?       github.com/fermyon/spin-operator/pkg/spinapp    [no test files]
ok      github.com/fermyon/spin-operator/internal/controller    7.827s  coverage: 34.3% of statements
fatal error: concurrent map iteration and map write

goroutine 43 [running]:
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.objectGVKs(0x14000331570, {0x1039c4e10?, 0x14000b82310})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/conversion/conversion.go:291 +0x130
sigs.k8s.io/controller-runtime/pkg/webhook/conversion.IsConvertible(0x14000331570?, {0x1039c4e10?, 0x14000b82310})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/conversion/conversion.go:227 +0x34
sigs.k8s.io/controller-runtime/pkg/envtest.modifyConversionWebhooks({0x14000818960, 0x3, 0x10?}, 0x14000331570, {{0x1400028aa80, 0x1, 0x1}, {0x1400032a6b8, 0x1, 0x1}, ...})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/crd.go:357 +0x110
sigs.k8s.io/controller-runtime/pkg/envtest.InstallCRDs(_, {0x14000331570, {0x14000111360, 0x1, 0x1}, {0x14000818960, 0x3, 0x3}, 0x1, 0x2540be400, ...})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/crd.go:101 +0xf4
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start(0x14000180500)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/server.go:282 +0x980
github.com/fermyon/spin-operator/internal/webhook.setupEnvTest(0x1400031a9c0)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:55 +0x308
github.com/fermyon/spin-operator/internal/webhook.TestCreateSpinAppWithNoExecutor(0x14000320401?)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:131 +0x28
testing.tRunner(0x1400031a9c0, 0x1039b5d60)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 1 [chan receive]:
testing.tRunner.func1()
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1561 +0x434
testing.tRunner(0x1400031a820, 0x14000193c28)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1601 +0x124
testing.runTests(0x1400026c280?, {0x1045dbf20, 0x8, 0x8}, {0x40?, 0x10387f5c0?, 0x1045f5b20?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x3b4
testing.(*M).Run(0x1400026c280)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0x538
main.main()
    _testmain.go:95 +0x1c8

goroutine 44 [select]:
github.com/stretchr/testify/assert.Eventually({0x10bb09318, 0x1400031ad00}, 0x14000202be0, 0x14000026000?, 0x14000202be0?, {0x0, 0x0, 0x0})
    /Users/caleb/go/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1847 +0x14c
github.com/stretchr/testify/require.Eventually({0x1039c1bb8, 0x1400031ad00}, 0x1400057bb98?, 0x2?, 0x2?, {0x0, 0x0, 0x0})
    /Users/caleb/go/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require.go:401 +0x94
github.com/fermyon/spin-operator/internal/webhook.startWebhookServer(0x1400031ad00, 0x140000afcc0)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:117 +0x428
github.com/fermyon/spin-operator/internal/webhook.TestCreateSpinAppWithSingleExecutor(0x14000320401?)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:151 +0x3c
testing.tRunner(0x1400031ad00, 0x1039b5d68)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 45 [select]:
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start(0x140001785a0, {0x0?, 0x0}, {0x0?, 0x0})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:183 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/testing/controlplane.(*APIServer).Start(0x1400038a000)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/controlplane/apiserver.go:165 +0x4c
sigs.k8s.io/controller-runtime/pkg/internal/testing/controlplane.(*ControlPlane).Start(0x1400022c000)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/controlplane/plane.go:67 +0x134
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane(0x1400022c000)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/server.go:313 +0x94
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start(0x1400022c000)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/envtest/server.go:244 +0x380
github.com/fermyon/spin-operator/internal/webhook.setupEnvTest(0x1400031b040)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:55 +0x308
github.com/fermyon/spin-operator/internal/webhook.TestCreateSpinAppWithMultipleExecutors(0x14000320401?)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:186 +0x2c
testing.tRunner(0x1400031b040, 0x1039b5d58)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 46 [runnable]:
k8s.io/apimachinery/pkg/runtime/serializer.CodecFactory.UniversalDecoder({0x14000331570, {0x1039bde60, 0x14000303068}, {0x140003428c0, 0x3, 0x4}, {0x1039cc170, 0x14000323f40}}, {0x0?, 0x102f3a578?, ...})
    /Users/caleb/go/pkg/mod/k8s.io/apimachinery@v0.29.1/pkg/runtime/serializer/codec_factory.go:289 +0xf0
k8s.io/client-go/discovery.setDiscoveryDefaults(0x1400094f860)
    /Users/caleb/go/pkg/mod/k8s.io/client-go@v0.29.1/discovery/discovery_client.go:710 +0xc4
k8s.io/client-go/discovery.NewDiscoveryClientForConfigAndClient(0x14000c1a001?, 0x1038e3420?)
    /Users/caleb/go/pkg/mod/k8s.io/client-go@v0.29.1/discovery/discovery_client.go:739 +0x54
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0x140003ae900?, 0x14000c12180?)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/client/apiutil/restmapper.go:40 +0x20
sigs.k8s.io/controller-runtime/pkg/client.newClient(0x1400031b300?, {0x14000c04900, 0x14000331570, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/client/client.go:159 +0x1c0
sigs.k8s.io/controller-runtime/pkg/client.New(0x1039c1bb8?, {0x0, 0x14000331570, {0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/client/client.go:110 +0x54
github.com/fermyon/spin-operator/internal/webhook.setupEnvTest(0x1400031b380)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:69 +0x460
github.com/fermyon/spin-operator/internal/webhook.TestCreateInvalidSpinApp(0x14000320401?)
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:231 +0x28
testing.tRunner(0x1400031b380, 0x1039b5d50)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0xe8
created by testing.(*T).Run in goroutine 1
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x33c

goroutine 72 [syscall]:
syscall.syscall6(0x104bb0a68?, 0x14000003617?, 0x1400008ee28?, 0x10230c5c8?, 0x1400008ee28?, 0x9000010101c654?, 0x10b9ead50?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x1400008ee88?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x18?, 0x1400008eec4, 0x3?, 0x8?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140005ac1b0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x1400017c9a0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 45
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 164 [sleep]:
time.Sleep(0x5f5e100)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/time.go:195 +0x10c
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.pollURLUntilOK({{0x1033cb2ab, 0x5}, {0x0, 0x0}, 0x0, {0x14000336fe0, 0xf}, {0x1033cf6d5, 0x8}, {0x0, ...}, ...}, ...)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:239 +0xd8
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 45
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:163 +0x270

goroutine 15 [syscall]:
syscall.syscall6(0x0?, 0x0?, 0x0?, 0x0?, 0x0?, 0x100010000?, 0x104f551b0?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140002fa688?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x0?, 0x140002fa6c4, 0x0?, 0x0?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140004e0000)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x14000344420)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 46
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 16 [syscall]:
syscall.syscall6(0x104bb0108?, 0x17?, 0x0?, 0x0?, 0x0?, 0x90000101010000?, 0x104f551b0?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140002f9688?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x0?, 0x140002f96c4, 0x0?, 0x0?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140003460f0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140006a2000)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 44
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 114 [syscall]:
syscall.syscall6(0x104bb05b8?, 0x17?, 0x1039e4e28?, 0x103818ea0?, 0x1039e4e28?, 0x90000101014620?, 0x104f499a0?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140002fd688?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x1033eb16f?, 0x140002fd6c4, 0x1033d18f5?, 0xa?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x1400004a0f0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x1400020e2c0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 43
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 324 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x140003210e0)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 119 [syscall]:
syscall.syscall6(0x104bb05b8?, 0x1039c8317?, 0x1039bdb00?, 0x1039cb900?, 0x0?, 0x90000101010000?, 0x10b9ab290?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x14000020e88?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x1039c4578?, 0x14000020ec4, 0x0?, 0x1039c4b90?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x140004962d0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140005a4160)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 43
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 165 [syscall]:
syscall.syscall6(0x1400045ae00?, 0x10230c824?, 0x1038189a0?, 0x10230cb68?, 0xffffffffffffffff?, 0xffff00010001ffff?, 0x104f499a0?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x1400045ae88?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x1400045aec0?, 0x1400045aec4, 0x1400045afa0?, 0x2?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x1400004a900)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x1400020eb00)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 45
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 141 [syscall]:
syscall.syscall6(0x14000088e00?, 0x10230c824?, 0x1038189a0?, 0x10230cb68?, 0xffffffffffffffff?, 0xffff00010001ffff?, 0x10b9ab290?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x14000088e88?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x14000088ec0?, 0x14000088ec4, 0x14000088fa0?, 0x2?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x14000496930)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140005a4840)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 46
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 134 [syscall]:
syscall.syscall6(0x104bb13c8?, 0x17?, 0x1038189a0?, 0x0?, 0x5ac?, 0x90000101017530?, 0x10b9ab290?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x140004c5688?, 0x1023e9768?, 0x90?, 0x10394f260?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x140004c56c0?, 0x140004c56c4, 0x140004c57a0?, 0x2?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x14000496450)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec.go:134
os/exec.(*Cmd).Wait(0x140005a42c0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/os/exec/exec.go:890 +0x38
sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:175 +0x5c
created by sigs.k8s.io/controller-runtime/pkg/internal/testing/process.(*State).Start in goroutine 44
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/internal/testing/process/process.go:173 +0x2f8

goroutine 322 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/cache/internal.(*Informers).Start(0x14000541400, {0x1039d5c90, 0x1400036b0e0})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/cache/internal/informers.go:211 +0x40
sigs.k8s.io/controller-runtime/pkg/cluster.(*cluster).Start(0x8?, {0x1039d5c90?, 0x1400036b0e0?})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/cluster/internal.go:104 +0x70
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0x14000202c00)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:223 +0xd0
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 273
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:207 +0x204

goroutine 327 [select]:
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Watch(0x140005180f0)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:126 +0x80
created by sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Start in goroutine 308
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:113 +0x1c4

goroutine 65 [IO wait]:
internal/poll.runtime_pollWait(0x10b9c4ca0, 0x72)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x140005f9f00?, 0x14000790000?, 0x0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140005f9f00, {0x14000790000, 0xa000, 0xa000})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x140005f9f00, {0x14000790000?, 0x140006c3828?, 0x1026549cc?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140004a4018, {0x14000790000?, 0x140006c3748?, 0x102313f0c?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x34
crypto/tls.(*atLeastReader).Read(0x1400000fe48, {0x14000790000?, 0x1400000fe48?, 0x0?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:805 +0x40
bytes.(*Buffer).ReadFrom(0x1400053c2a8, {0x1039bd008, 0x1400000fe48})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x1400053c000, {0x10bc0f658?, 0x140004a4018}, 0x9fb9?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:827 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x1400053c000, 0x0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:625 +0x1e4
crypto/tls.(*Conn).readRecord(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0x1400053c000, {0x1400082b000, 0x1000, 0x1027df87c?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:1369 +0x168
bufio.(*Reader).Read(0x1400051c8a0, {0x14000236580, 0x9, 0x1045cb310?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:244 +0x1b4
io.ReadAtLeast({0x1039bca88, 0x1400051c8a0}, {0x14000236580, 0x9, 0x9}, 0x9)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:335 +0xa0
io.ReadFull(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x14000236580, 0x9, 0x172604d7?}, {0x1039bca88?, 0x1400051c8a0?})
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x14000236540)
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:498 +0x78
golang.org/x/net/http2.(*clientConnReadLoop).run(0x140006c3f88)
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2275 +0xf8
golang.org/x/net/http2.(*ClientConn).readLoop(0x14000828000)
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2170 +0x5c
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 64
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:821 +0xabc

goroutine 302 [select]:
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).Start(0x140006a7380, {0x1039d5bb0, 0x10462bd80})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/internal.go:444 +0x768
github.com/fermyon/spin-operator/internal/webhook.startWebhookServer.func1()
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:110 +0x48
created by github.com/fermyon/spin-operator/internal/webhook.startWebhookServer in goroutine 44
    /Users/caleb/fermyon/spin-operator/internal/webhook/admission_test.go:109 +0x2f8

goroutine 303 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000320ea0)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 304 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000320f30)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 305 [IO wait]:
internal/poll.runtime_pollWait(0x10b9c48c0, 0x72)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x1400032ee80?, 0xd0000001003420?, 0x0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x1400032ee80)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:611 +0x250
net.(*netFD).accept(0x1400032ee80)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000202d80)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock_posix.go:152 +0x28
net.(*TCPListener).Accept(0x14000202d80)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/tcpsock.go:315 +0x2c
crypto/tls.(*listener).Accept(0x140005e4de0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/tls.go:66 +0x30
net/http.(*Server).Serve(0x140002d0000, {0x1039c7fd0, 0x140005e4de0})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/http/server.go:3056 +0x2b8
sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start(0x14000225860, {0x1039d5c90?, 0x1400036b090})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:263 +0x774
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0x14000202520)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:223 +0xd0
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 304
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:207 +0x204

goroutine 273 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000320fc0)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 302
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 326 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile(0x14000321050)
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:183 +0x3c
created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).Start.func1 in goroutine 325
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/manager/runnable_group.go:136 +0xb0

goroutine 307 [syscall]:
syscall.syscall6(0x14000078bb8?, 0x10b9b23c8?, 0x88?, 0x104bb0130?, 0x14000078c28?, 0x1023729b4?, 0x104bb0108?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/sys_darwin.go:45 +0x68
golang.org/x/sys/unix.kevent(0x2000?, 0x14000026000?, 0x14000078c68?, 0x0?, 0x14000078c58?, 0x10240e63c?)
    /Users/caleb/go/pkg/mod/golang.org/x/sys@v0.16.0/unix/zsyscall_darwin_arm64.go:275 +0x54
golang.org/x/sys/unix.Kevent(0x14000078d00?, {0x0?, 0x14000078c98?, 0x10240e74c?}, {0x14000078e60?, 0x102313624?, 0x14000078cb8?}, 0x102313638?)
    /Users/caleb/go/pkg/mod/golang.org/x/sys@v0.16.0/unix/syscall_bsd.go:397 +0x40
github.com/fsnotify/fsnotify.(*Watcher).read(0x14000078d48?, {0x14000078e60?, 0x14000078d68?, 0xa})
    /Users/caleb/go/pkg/mod/github.com/fsnotify/fsnotify@v1.7.0/backend_kqueue.go:777 +0x48
github.com/fsnotify/fsnotify.(*Watcher).readEvents(0x14000204f50)
    /Users/caleb/go/pkg/mod/github.com/fsnotify/fsnotify@v1.7.0/backend_kqueue.go:547 +0x94
created by github.com/fsnotify/fsnotify.NewBufferedWatcher in goroutine 305
    /Users/caleb/go/pkg/mod/github.com/fsnotify/fsnotify@v1.7.0/backend_kqueue.go:184 +0x1fc

goroutine 308 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/certwatcher.(*CertWatcher).Start(0x140005180f0, {0x1039d5c90, 0x1400036b090})
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/certwatcher/certwatcher.go:118 +0x210
sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start.func1()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:214 +0x28
created by sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start in goroutine 305
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:213 +0x2ac

goroutine 309 [chan receive]:
sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start.func2()
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:248 +0x48
created by sigs.k8s.io/controller-runtime/pkg/webhook.(*DefaultServer).Start in goroutine 305
    /Users/caleb/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.0/pkg/webhook/server.go:247 +0x6ac

goroutine 365 [IO wait]:
internal/poll.runtime_pollWait(0x10b9c49b8, 0x72)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x140001c1380?, 0x14000520000?, 0x0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140001c1380, {0x14000520000, 0xa000, 0xa000})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x140001c1380, {0x14000520000?, 0x1400001b828?, 0x1026549cc?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140004a4058, {0x14000520000?, 0x1400001b748?, 0x102313d94?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/net/net.go:179 +0x34
crypto/tls.(*atLeastReader).Read(0x14000b12018, {0x14000520000?, 0x14000b12018?, 0x140005fdd40?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:805 +0x40
bytes.(*Buffer).ReadFrom(0x140004ce2a8, {0x1039bd008, 0x14000b12018})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/bytes/buffer.go:211 +0x90
crypto/tls.(*Conn).readFromUntil(0x140004ce000, {0x10bc0f658?, 0x140004a4058}, 0x9fb8?)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:827 +0xd0
crypto/tls.(*Conn).readRecordOrCCS(0x140004ce000, 0x0)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:625 +0x1e4
crypto/tls.(*Conn).readRecord(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0x140004ce000, {0x1400048f000, 0x1000, 0x1027df87c?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/crypto/tls/conn.go:1369 +0x168
bufio.(*Reader).Read(0x1400060d740, {0x1400038a2e0, 0x9, 0x1045cb310?})
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/bufio/bufio.go:244 +0x1b4
io.ReadAtLeast({0x1039bca88, 0x1400060d740}, {0x1400038a2e0, 0x9, 0x9}, 0x9)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:335 +0xa0
io.ReadFull(...)
    /opt/homebrew/Cellar/go/1.21.6/libexec/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0x1400038a2e0, 0x9, 0x18139bd4?}, {0x1039bca88?, 0x1400060d740?})
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x1400038a2a0)
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/frame.go:498 +0x78
golang.org/x/net/http2.(*clientConnReadLoop).run(0x1400001bf88)
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2275 +0xf8
golang.org/x/net/http2.(*ClientConn).readLoop(0x140008d2000)
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:2170 +0x5c
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 364
    /Users/caleb/go/pkg/mod/golang.org/x/net@v0.19.0/http2/transport.go:821 +0xabc
FAIL    github.com/fermyon/spin-operator/internal/webhook   3.849s
FAIL
make: *** [test] Error 1
rajatjindal commented 8 months ago

I tried running the tests a few times to reproduce this issue, but unable to. Is there a specific command that triggers this behavior?

endocrimes commented 8 months ago

I think this is actually because we use the shared scheme in the testenv client:

https://github.com/spinkube/spin-operator/blob/main/internal/webhook/admission_test.go#L63-L71

(I wasn't super expecting parallel integration tests - we'll need to switch it to making a new scheme, add in default types, and our types)

calebschoepp commented 8 months ago

I tried running the tests a few times to reproduce this issue, but unable to. Is there a specific command that triggers this behavior?

I was getting locally just running make test. Might be really rare or something that is specific to my machine.

bacongobbler commented 8 months ago

Removing from the initial milestone - looks like something we can fix async. I don't think the bug actually affects anything in prod, but please correct me if I'm wrong on that.

calebschoepp commented 8 months ago

Removing from the initial milestone - looks like something we can fix async. I don't think the bug actually affects anything in prod, but please correct me if I'm wrong on that.

Correct, this has no impact on prod deployments.

calebschoepp commented 8 months ago

@endocrimes does #134 fix this?

endocrimes commented 8 months ago

Yep