search

k0sproject / k0s

k0s - The Zero Friction Kubernetes
https://docs.k0sproject.io
Other
3.12k stars 353 forks source link

fatal error: concurrent map read and map write #4636

Closed emosbaugh closed 2 weeks ago

emosbaugh commented 2 weeks ago

Before creating an issue, make sure you've checked the following:

Platform

No response

Version

v1.29.5+k0s.0

Sysinfo

`k0s sysinfo` 
Machine ID: "8e7e0fcbc82ee318fa1847250e5f843b994314ff9cca59cbb568b251e9fe2e4e" (from machine) (pass)
Total memory: 15.6 GiB (pass)
Disk space available for /var/lib/k0s: 21.6 GiB (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.5.0-1021-azure (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: exec: "modprobe": executable file not found in $PATH (warning)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: no kernel config found (warning)
  CONFIG_NAMESPACES: Namespaces support: no kernel config found (warning)
  CONFIG_NET: Networking support: no kernel config found (warning)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: no kernel config found (warning)
  CONFIG_PROC_FS: /proc file system support: no kernel config found (warning)

What happened?

startSubControllerRoutine calls AddToScheme but this should never happen in concurrent code.

https://github.com/k0sproject/k0s/blob/v1.29.5%2Bk0s.0/pkg/autopilot/controller/root_controller.go#L243-L245 https://github.com/k0sproject/k0s/blob/v1.29.5%2Bk0s.0/pkg/autopilot/controller/root_controller.go#L163

Panic is in this goroutine:

https://github.com/k0sproject/k0s/blob/v1.29.5%2Bk0s.0/pkg/leaderelection/lease_pool.go#L225-L227

I'm not running the latest but I dont think this code has changed since.

Similar issues in other notable projects: https://github.com/helm/helm/issues/6566 https://github.com/rook/rook/issues/4350

Steps to reproduce

  1. Its a race condition

Expected behavior

No response

Actual behavior

No response

Screenshots and logs

k0scontroller-logs.txt

Additional context

Relevant parts of the stack trace:

Jun 14 14:52:11 node-16d20-00 k0s[944]: fatal error: concurrent map read and map write
Jun 14 14:52:11 node-16d20-00 k0s[944]: goroutine 3684 [running]:
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*Scheme).ObjectKinds(0xc0003b41c0, {0x3ffef98?, 0xc001a29860})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/scheme.go:263 +0xbd
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*parameterCodec).EncodeParameters(0x5eb6020, {0x3ffef98, 0xc001a29860}, {{0x38da8f5, 0x13}, {0x3887ae8, 0x2}})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/codec.go:190 +0x62
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/rest.(*Request).SpecificallyVersionedParams(0xc0010e1440, {0x3ffef98?, 0xc001a29860?}, {0x3fff268?, 0x5eb6020?}, {{0x38da8f5?, 0x4febf89?}, {0x3887ae8?, 0x412c65?}})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/rest/request.go:377 +0x75
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/rest.(*Request).VersionedParams(...)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/rest/request.go:370
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/kubernetes/typed/coordination/v1.(*leases).Get(0xc0017bf7a0, {0x4027e08, 0xc0018e2e10}, {0xc000b6e918, 0x18}, {{{0x0, 0x0}, {0x0, 0x0}}, {0x0, ...}})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/kubernetes/typed/coordination/v1/lease.go:77 +0x145
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/tools/leaderelection/resourcelock.(*LeaseLock).Get(0xc0019d9d40, {0x4027e08, 0xc0018e2e10})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/tools/leaderelection/resourcelock/leaselock.go:42 +0x89
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/tools/leaderelection.(*LeaderElector).tryAcquireOrRenew(0xc002068000, {0x4027e08, 0xc0018e2e10})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/tools/leaderelection/leaderelection.go:329 +0x173
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire.func1()
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/tools/leaderelection/leaderelection.go:252 +0x5a
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/util/wait/backoff.go:226 +0x33
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001a0cfc0?, {0x3fee740, 0xc001a29830}, 0x1, 0xc001a0cfc0)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/util/wait/backoff.go:227 +0xaf
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0018e2e10?, 0x12a05f200, 0x3ff3333333333333, 0xfc?, 0x32ad540?)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/util/wait/backoff.go:204 +0x7f
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/tools/leaderelection.(*LeaderElector).acquire(0xc002068000, {0x4027e08?, 0xc001aa6aa0?})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/tools/leaderelection/leaderelection.go:251 +0x205
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc002068000, {0x4027e08, 0xc001aa6aa0})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/client-go@v0.29.5/tools/leaderelection/leaderelection.go:208 +0x56
Jun 14 14:52:11 node-16d20-00 k0s[944]: github.com/k0sproject/k0s/pkg/leaderelection.(*LeasePool).Watch.func3()
Jun 14 14:52:11 node-16d20-00 k0s[944]:         github.com/k0sproject/k0s/pkg/leaderelection/lease_pool.go:227 +0x3f
Jun 14 14:52:11 node-16d20-00 k0s[944]: created by github.com/k0sproject/k0s/pkg/leaderelection.(*LeasePool).Watch in goroutine 3683
Jun 14 14:52:11 node-16d20-00 k0s[944]:         github.com/k0sproject/k0s/pkg/leaderelection/lease_pool.go:225 +0x42a
Jun 14 14:52:11 node-16d20-00 k0s[944]: goroutine 3686 [runnable]:
Jun 14 14:52:11 node-16d20-00 k0s[944]: type:.eq.k8s.io/apimachinery/pkg/runtime/schema.GroupVersionKind(0xc001b54860?, 0xc00084bce8?)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         <autogenerated>:1 +0xa5
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypeWithName(0xc0003b41c0, {{0x0, 0x0}, {0x3887ae8, 0x2}, {0x2f23920, 0xf}}, {0x3fff218?, 0xc001aa7f90})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/scheme.go:174 +0x252
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*Scheme).AddKnownTypes(0xc0003b41c0, {{0x0?, 0x474093?}, {0x3887ae8?, 0xc001b54a30?}}, {0xc001b54bc0?, 0x5, 0xc000dd3400?})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/scheme.go:148 +0x176
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*Scheme).AddUnversionedTypes(0xc0003b41c0, {{0x0?, 0x11?}, {0x3887ae8?, 0xa?}}, {0xc001b54bc0?, 0x5, 0x5?})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/scheme.go:124 +0xaf
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/apis/meta/v1.AddToGroupVersion(0xc0003b41c0?, {{0x38cd7dd?, 0xc000837380?}, {0x389ab0a?, 0x0?}})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/apis/meta/v1/register.go:75 +0x335
Jun 14 14:52:11 node-16d20-00 k0s[944]: github.com/k0sproject/k0s/pkg/apis/k0s/v1beta1.init.0.(*Builder).Register.func1(0xc001b54d10?)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         sigs.k8s.io/controller-runtime@v0.17.0/pkg/scheme/scheme.go:72 +0x5a
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(...)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/scheme_builder.go:29
Jun 14 14:52:11 node-16d20-00 k0s[944]: sigs.k8s.io/controller-runtime/pkg/scheme.(*Builder).AddToScheme(0xc0009b4580?, 0x0?)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         sigs.k8s.io/controller-runtime@v0.17.0/pkg/scheme/scheme.go:86 +0x56
Jun 14 14:52:11 node-16d20-00 k0s[944]: k8s.io/apimachinery/pkg/runtime.(*SchemeBuilder).AddToScheme(...)
Jun 14 14:52:11 node-16d20-00 k0s[944]:         k8s.io/apimachinery@v0.29.5/pkg/runtime/scheme_builder.go:29
Jun 14 14:52:11 node-16d20-00 k0s[944]: github.com/k0sproject/k0s/pkg/autopilot/controller.(*rootController).startSubControllerRoutine(0xc00110c9c0, {0x4027e08, 0xc001aa6b90}, 0xc000512380, {0x389b536, 0x7})
Jun 14 14:52:11 node-16d20-00 k0s[944]:         github.com/k0sproject/k0s/pkg/autopilot/controller/root_controller.go:163 +0x34b
Jun 14 14:52:11 node-16d20-00 k0s[944]: github.com/k0sproject/k0s/pkg/autopilot/controller.(*rootController).startSubControllers.func1()
Jun 14 14:52:11 node-16d20-00 k0s[944]:         github.com/k0sproject/k0s/pkg/autopilot/controller/root_controller.go:245 +0xa8
Jun 14 14:52:11 node-16d20-00 k0s[944]: golang.org/x/sync/errgroup.(*Group).Go.func1()
Jun 14 14:52:11 node-16d20-00 k0s[944]:         golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
Jun 14 14:52:11 node-16d20-00 k0s[944]: created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1710
Jun 14 14:52:11 node-16d20-00 k0s[944]:         golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96
jnummelin commented 2 weeks ago

I believe this is fixed in #4537

emosbaugh commented 2 weeks ago

Yes it is thank you. closing