search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.62k stars 531 forks source link

Incompatibility with Kubectl Version 1.26.3 #7110

Closed scrayos closed 1 year ago

scrayos commented 1 year ago

Bug Report

Description

After Updating to Talos Linux 1.4.0 (Kubernetes 1.27.1), Kubectl Version 1.26.3 can no longer communicate with the cluster. I tested it with kubectl get pods but other commands don't work as well. Additionally there are two containers that communicate with the Kubernetes API that also started failing.

Everything was working fine until I updated to last worker node. So the first 5 nodes worked just fine. I don't know if this is a Talos Linux Problem or a Kubernetes problem, but the version difference should be supported by the Kubernetes version skew policy.

Logs

Rook-Ceph-Operator:

2023-04-19 11:08:22.491914 I | rookcmd: starting Rook v1.11.3 with arguments '/usr/local/bin/rook ceph operator'
2023-04-19 11:08:22.491976 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account=
2023-04-19 11:08:22.491979 I | cephcmd: starting Rook-Ceph operator
2023-04-19 11:08:22.706361 I | cephcmd: base ceph version inside the rook operator image is "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)"
2023-04-19 11:08:22.813675 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2023-04-19 11:08:22.813690 I | operator: watching all namespaces for Ceph CRs
2023-04-19 11:08:22.813755 I | operator: setting up schemes
2023-04-19 11:08:22.815942 I | operator: setting up the controller-runtime manager
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12edd10]

goroutine 81 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:88
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc000625068, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:69 +0x5f0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc000624018, 0x15}, {0xc00015e260, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/aggregated_discovery.go:35 +0x2f8
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0x25fea6c?)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:310 +0x47c
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xc0001285a0?)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:198 +0x5c
k8s.io/client-go/discovery.ServerGroupsAndResources({0x26299d0, 0xc000904ff0})
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:392 +0x59
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources.func1()
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:356 +0x25
k8s.io/client-go/discovery.withRetries(0x2, 0xc0005a50d0)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:621 +0x72
k8s.io/client-go/discovery.(*DiscoveryClient).ServerGroupsAndResources(0x0?)
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/discovery/discovery_client.go:355 +0x3a
k8s.io/client-go/restmapper.GetAPIGroupResources({0x26299d0?, 0xc000904ff0?})
    /home/runner/go/pkg/mod/k8s.io/client-go@v0.26.1/restmapper/discovery.go:148 +0x42
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper.func1()
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/client/apiutil/dynamicrestmapper.go:86 +0x25
sigs.k8s.io/controller-runtime/pkg/client/apiutil.(*dynamicRESTMapper).setStaticMapper(...)
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/client/apiutil/dynamicrestmapper.go:119
sigs.k8s.io/controller-runtime/pkg/client/apiutil.NewDynamicRESTMapper(0xc0008eecf0?, {0x0, 0x0, 0x9335f667369e3001?})
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/client/apiutil/dynamicrestmapper.go:99 +0x176
sigs.k8s.io/controller-runtime/pkg/cluster.setOptionsDefaults.func1(0xc0004f9030?)
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/cluster/cluster.go:217 +0x25
sigs.k8s.io/controller-runtime/pkg/cluster.New(0xc000888d80, {0xc000999a58, 0x1, 0x0?})
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/cluster/cluster.go:159 +0x18d
sigs.k8s.io/controller-runtime/pkg/manager.New(_, {0xc0004f9030, 0x0, 0x0, {{0x2623510, 0xc0001234c0}, 0x0}, 0x0, {0x0, 0x0}, ...})
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.1/pkg/manager/manager.go:351 +0xf9
github.com/rook/rook/pkg/operator/ceph.(*Operator).startCRDManager(0xc000591260, {0x261fbc0?, 0xc000123f00}, 0xc0000c6c60?)
    /home/runner/work/rook/rook/pkg/operator/ceph/cr_manager.go:164 +0x275
created by github.com/rook/rook/pkg/operator/ceph.(*Operator).runCRDManager
    /home/runner/work/rook/rook/pkg/operator/ceph/operator.go:141 +0x185

Kubectl:

kubectl get pods
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x55c08ab1d5b0]

goroutine 1 [running]:
k8s.io/client-go/discovery.convertAPIResource(...)
    vendor/k8s.io/client-go/discovery/aggregated_discovery.go:114
k8s.io/client-go/discovery.convertAPIGroup({{{0x0, 0x0}, {0x0, 0x0}}, {{0xc0002a7200, 0x15}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    vendor/k8s.io/client-go/discovery/aggregated_discovery.go:95 +0x7d0
k8s.io/client-go/discovery.SplitGroupsAndResources({{{0xc0002a6000, 0x15}, {0xc00063a160, 0x1b}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...})
    vendor/k8s.io/client-go/discovery/aggregated_discovery.go:49 +0x318
k8s.io/client-go/discovery.(*DiscoveryClient).downloadAPIs(0xd4998570fe3a39d8?)
    vendor/k8s.io/client-go/discovery/discovery_client.go:328 +0x3de
k8s.io/client-go/discovery.(*DiscoveryClient).GroupsAndMaybeResources(0xffffffffffffffff?)
    vendor/k8s.io/client-go/discovery/discovery_client.go:203 +0x65
k8s.io/client-go/discovery/cached/memory.(*memCacheClient).refreshLocked(0xc000128ae0)
    vendor/k8s.io/client-go/discovery/cached/memory/memcache.go:245 +0x57
k8s.io/client-go/discovery/cached/memory.(*memCacheClient).GroupsAndMaybeResources(0xc000128ae0)
    vendor/k8s.io/client-go/discovery/cached/memory/memcache.go:144 +0xcf
k8s.io/client-go/discovery/cached/memory.(*memCacheClient).ServerGroups(0xc0000d3720?)
    vendor/k8s.io/client-go/discovery/cached/memory/memcache.go:170 +0x19
k8s.io/client-go/discovery/cached/disk.(*CachedDiscoveryClient).ServerGroups(0xc0000d3720)
    vendor/k8s.io/client-go/discovery/cached/disk/cached_discovery.go:118 +0x172
k8s.io/client-go/discovery.ServerGroupsAndResources({0x55c08bb71b70, 0xc0000d3720})
    vendor/k8s.io/client-go/discovery/discovery_client.go:418 +0xca
k8s.io/client-go/discovery/cached/disk.(*CachedDiscoveryClient).ServerGroupsAndResources(0x0?)
    vendor/k8s.io/client-go/discovery/cached/disk/cached_discovery.go:101 +0x25
k8s.io/client-go/restmapper.discoveryCategoryExpander.Expand({{0x55c08bb71b70?, 0xc0000d3720?}}, {0x7ffc3473bff6, 0x4})
    vendor/k8s.io/client-go/restmapper/category_expansion.go:61 +0x52
k8s.io/cli-runtime/pkg/resource.(*Builder).ReplaceAliases(0xc0004dd4a0, {0x7ffc3473bff6?, 0xc000601000?})
    vendor/k8s.io/cli-runtime/pkg/resource/builder.go:645 +0x20a
k8s.io/cli-runtime/pkg/resource.(*Builder).ResourceTypeOrNameArgs(0xc0004dd4a0, 0x1, {0xc0001d7c30?, 0x55c08b1bc208?, 0xc000d97c30?})
    vendor/k8s.io/cli-runtime/pkg/resource/builder.go:609 +0x185
k8s.io/kubectl/pkg/cmd/get.(*GetOptions).Run(0xc000144480, {0x55c08bb74280?, 0xc0001650c0?}, 0xc00046c600?, {0xc0001d7c30, 0x1, 0x1})
    vendor/k8s.io/kubectl/pkg/cmd/get/get.go:485 +0x216
k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc00046c600?, {0xc0001d7c30, 0x1, 0x1})
    vendor/k8s.io/kubectl/pkg/cmd/get/get.go:173 +0xb2
github.com/spf13/cobra.(*Command).execute(0xc00046c600, {0xc0001d7b30, 0x1, 0x1})
    vendor/github.com/spf13/cobra/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xc00046c000)
    vendor/github.com/spf13/cobra/command.go:1040 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
    vendor/github.com/spf13/cobra/command.go:968
k8s.io/component-base/cli.run(0xc00046c000)
    vendor/k8s.io/component-base/cli/run.go:146 +0x317
k8s.io/component-base/cli.RunNoErrOutput(...)
    vendor/k8s.io/component-base/cli/run.go:84
main.main()
    cmd/kubectl/kubectl.go:30 +0x1e

Environment

smira commented 1 year ago

This issue doesn't seem to be related to Talos. Talos runs vanilla Kubernetes. Might be a bug in kubectl, but once again there's nothing on Talos side we can do about it.

I have no issues with kubectl 1.26.4 and Kubernetes 1.27.1. Also it makes sense to use latest kubectl 1.26.x in case if it's a bug that got already fixed.