search

k0sproject / k0smotron

k0smotron
https://docs.k0smotron.io/
Other
461 stars 42 forks source link

ControlPlane manager throwing segfaults (control plane does come up) #644

Closed Silvest89 closed 1 month ago

Silvest89 commented 1 month ago

This is how my control plane looks like. The cluster does come up fine however the controller gets quite a few segfaults(which causes restarts)

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: K0smotronControlPlane
metadata:
  name: silvenia
spec:
  replicas: 3
  version: v1.30.2
  persistence:
    type: pvc
    persistentVolumeClaim:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  service:
    type: NodePort
    apiPort: 30450
    konnectivityPort: 30451
  kineDataSourceSecretName: database-credentials
  externalAddress: *****
  k0sConfig:
    apiVersion: k0s.k0sproject.io/v1beta1
    kind: ClusterConfig
    metadata:
      name: k0s
    spec:
      network:
        provider: custom
2024-07-06T20:05:22Z    INFO    setup   starting manager
2024-07-06T20:05:22Z    INFO    controller-runtime.metrics  Starting metrics server
2024-07-06T20:05:22Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-07-06T20:05:22Z    INFO    controller-runtime.metrics  Serving metrics server  {"bindAddress": "127.0.0.1:8080", "secure": false}
I0706 20:05:22.598624       1 leaderelection.go:250] attempting to acquire leader lease k0smotron/f6ddf68f34963ade25bce356fc63d572.k0smotron.io...
I0706 20:05:38.394681       1 leaderelection.go:260] successfully acquired lease k0smotron/f6ddf68f34963ade25bce356fc63d572.k0smotron.io
2024-07-06T20:05:38Z    DEBUG   events  k0smotron-controller-manager-control-plane-584bf855bc-c2kn6_69522e3d-9476-4822-b9d9-5cf3522157b8 became leader  {"type": "Normal", "object": {"kind":"Lease","namespace":"k0smotron","name":"f6ddf68f34963ade25bce356fc63d572.k0smotron.io","uid":"68c4b962-1a46-47d2-8677-45245e22d493","apiVersion":"coordination.k8s.io/v1","resourceVersion":"315387"}, "reason": "LeaderElection"}
2024-07-06T20:05:38Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "source": "kind source: *v1beta1.Cluster"}
2024-07-06T20:05:38Z    INFO    Starting EventSource    {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "source": "kind source: *v1.StatefulSet"}
2024-07-06T20:05:38Z    INFO    Starting Controller {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster"}
2024-07-06T20:05:38Z    INFO    Starting EventSource    {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane", "source": "kind source: *v1beta1.K0smotronControlPlane"}
2024-07-06T20:05:38Z    INFO    Starting EventSource    {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane", "source": "kind source: *v1beta1.Cluster"}
2024-07-06T20:05:38Z    INFO    Starting Controller {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane"}
2024-07-06T20:05:38Z    INFO    Starting EventSource    {"controller": "jointokenrequest", "controllerGroup": "k0smotron.io", "controllerKind": "JoinTokenRequest", "source": "kind source: *v1beta1.JoinTokenRequest"}
2024-07-06T20:05:38Z    INFO    Starting Controller {"controller": "jointokenrequest", "controllerGroup": "k0smotron.io", "controllerKind": "JoinTokenRequest"}
2024-07-06T20:05:38Z    INFO    Starting EventSource    {"controller": "k0scontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0sControlPlane", "source": "kind source: *v1beta1.K0sControlPlane"}
2024-07-06T20:05:38Z    INFO    Starting Controller {"controller": "k0scontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0sControlPlane"}
2024-07-06T20:05:38Z    INFO    Starting workers    {"controller": "jointokenrequest", "controllerGroup": "k0smotron.io", "controllerKind": "JoinTokenRequest", "worker count": 1}
2024-07-06T20:05:38Z    INFO    Starting workers    {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "worker count": 1}
2024-07-06T20:05:38Z    INFO    Reconciling {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "Cluster": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "f09af156-e803-4820-acf0-9bec98f17ed4"}
2024-07-06T20:05:38Z    INFO    Reconciling services    {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "Cluster": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "f09af156-e803-4820-acf0-9bec98f17ed4"}
2024-07-06T20:05:38Z    INFO    Reconciling services    {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "Cluster": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "f09af156-e803-4820-acf0-9bec98f17ed4"}
2024-07-06T20:05:38Z    INFO    Starting workers    {"controller": "k0scontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0sControlPlane", "worker count": 1}
2024-07-06T20:05:38Z    INFO    Starting workers    {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane", "worker count": 1}
2024-07-06T20:05:38Z    INFO    Reconciling K0smotronControlPlane   {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane", "K0smotronControlPlane": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "24485c40-1d0d-4166-a3ab-7ba3b5bbdce7", "controlplane": {"name":"silvenia","namespace":"capi-cluster"}}
2024-07-06T20:05:38Z    INFO    Waiting for loadbalancer address    {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "Cluster": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "f09af156-e803-4820-acf0-9bec98f17ed4"}
2024-07-06T20:05:38Z    INFO    Reconciling configmap   {"controller": "cluster", "controllerGroup": "k0smotron.io", "controllerKind": "Cluster", "Cluster": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "f09af156-e803-4820-acf0-9bec98f17ed4"}
2024-07-06T20:05:38Z    INFO    Waiting for external address to be set  {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane", "K0smotronControlPlane": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "24485c40-1d0d-4166-a3ab-7ba3b5bbdce7", "cluster": "silvenia"}
E0706 20:05:38.720337       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 251 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1b190c0?, 0x35d9810})
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/runtime/runtime.go:75 +0x7c
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x18bd5e9?})
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/runtime/runtime.go:49 +0x78
panic({0x1b190c0?, 0x35d9810?})
    /usr/local/go/src/runtime/panic.go:914 +0x218
k8s.io/api/core/v1.(*ObjectReference).GroupVersionKind(...)
    /go/pkg/mod/k8s.io/api@v0.28.4/core/v1/objectreference.go:30
github.com/k0sproject/k0smotron/internal/controller/controlplane.(*K0smotronController).waitExternalAddress.func1({0x2269af0, 0x40003dc8c0})
    /workspace/internal/controller/controlplane/k0smotron_controlplane_controller.go:154 +0x150
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1(0x4000e33100?, {0x2269af0?, 0x40003dc8c0?})
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49 +0x58
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x2269af0, 0x40003dc8c0}, {0x2259e70?, 0x40005ceb60}, 0x1, 0x0, 0x400031ba00?)
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50 +0xa0
k8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout({0x2269a48?, 0x4000a066c0?}, 0x3b9aca00, 0x1f1c35a?, 0x26?, 0x0?)
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:48 +0xa8
github.com/k0sproject/k0smotron/internal/controller/controlplane.(*K0smotronController).waitExternalAddress(0x4000839f50, {0x2269a48, 0x4000a066c0}, 0x4000a97040, 0x4000534000?)
    /workspace/internal/controller/controlplane/k0smotron_controlplane_controller.go:136 +0x1a0
github.com/k0sproject/k0smotron/internal/controller/controlplane.(*K0smotronController).Reconcile(0x4000839f50, {0x2269a48, 0x4000a066c0}, {{{0x4000aba370, 0xc}, {0x4000aba368, 0x8}}})
    /workspace/internal/controller/controlplane/k0smotron_controlplane_controller.go:115 +0x408
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x226f168?, {0x2269a48?, 0x4000a066c0?}, {{{0x4000aba370?, 0xb?}, {0x4000aba368?, 0x0?}}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:119 +0x8c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0x4000936960, {0x2269a80, 0x4000757db0}, {0x1c2d600?, 0x400095c720?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:316 +0x2e4
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x4000936960, {0x2269a80, 0x4000757db0})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266 +0x198
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227 +0x74
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:223 +0x43c
2024-07-06T20:05:38Z    INFO    Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference    {"controller": "k0smotroncontrolplane", "controllerGroup": "controlplane.cluster.x-k8s.io", "controllerKind": "K0smotronControlPlane", "K0smotronControlPlane": {"name":"silvenia","namespace":"capi-cluster"}, "namespace": "capi-cluster", "name": "silvenia", "reconcileID": "24485c40-1d0d-4166-a3ab-7ba3b5bbdce7"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1254f40]

goroutine 251 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:116 +0x1a4
panic({0x1b190c0?, 0x35d9810?})
    /usr/local/go/src/runtime/panic.go:914 +0x218
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x18bd5e9?})
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/runtime/runtime.go:56 +0xe0
panic({0x1b190c0?, 0x35d9810?})
    /usr/local/go/src/runtime/panic.go:914 +0x218
k8s.io/api/core/v1.(*ObjectReference).GroupVersionKind(...)
    /go/pkg/mod/k8s.io/api@v0.28.4/core/v1/objectreference.go:30
github.com/k0sproject/k0smotron/internal/controller/controlplane.(*K0smotronController).waitExternalAddress.func1({0x2269af0, 0x40003dc8c0})
    /workspace/internal/controller/controlplane/k0smotron_controlplane_controller.go:154 +0x150
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext.func1(0x4000e33100?, {0x2269af0?, 0x40003dc8c0?})
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:49 +0x58
k8s.io/apimachinery/pkg/util/wait.loopConditionUntilContext({0x2269af0, 0x40003dc8c0}, {0x2259e70?, 0x40005ceb60}, 0x1, 0x0, 0x400031ba00?)
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/wait/loop.go:50 +0xa0
k8s.io/apimachinery/pkg/util/wait.PollUntilContextTimeout({0x2269a48?, 0x4000a066c0?}, 0x3b9aca00, 0x1f1c35a?, 0x26?, 0x0?)
    /go/pkg/mod/k8s.io/apimachinery@v0.28.4/pkg/util/wait/poll.go:48 +0xa8
github.com/k0sproject/k0smotron/internal/controller/controlplane.(*K0smotronController).waitExternalAddress(0x4000839f50, {0x2269a48, 0x4000a066c0}, 0x4000a97040, 0x4000534000?)
    /workspace/internal/controller/controlplane/k0smotron_controlplane_controller.go:136 +0x1a0
github.com/k0sproject/k0smotron/internal/controller/controlplane.(*K0smotronController).Reconcile(0x4000839f50, {0x2269a48, 0x4000a066c0}, {{{0x4000aba370, 0xc}, {0x4000aba368, 0x8}}})
    /workspace/internal/controller/controlplane/k0smotron_controlplane_controller.go:115 +0x408
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x226f168?, {0x2269a48?, 0x4000a066c0?}, {{{0x4000aba370?, 0xb?}, {0x4000aba368?, 0x0?}}})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:119 +0x8c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0x4000936960, {0x2269a80, 0x4000757db0}, {0x1c2d600?, 0x400095c720?})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:316 +0x2e4
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0x4000936960, {0x2269a80, 0x4000757db0})
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266 +0x198
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227 +0x74
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 141
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:223 +0x43c
makhov commented 1 month ago

Do you want to use k0smotron in Cluster API or standalone mode? controlplane.cluster.x-k8s.io/v1beta1.K0smotronControlPlane is an object for CAPI, it heavily relies on other CAPI objects, like in this case, it fails to find the infrastructureRef field of the cluster.x-k8s.io/v1beta1.Cluster object.

If you just want to deploy a control plane in pods without CAPI complexity, use k0smotron.io/v1beta1.Cluster object. Basically, you need to change only API version and kind in your config: 

apiVersion: k0smotron.io/v1beta1
kind: Cluster
metadata:
  name: silvenia
spec:
  replicas: 3
  version: v1.30.2
  persistence:
    type: pvc
    persistentVolumeClaim:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  service:
    type: NodePort
    apiPort: 30450
    konnectivityPort: 30451
  kineDataSourceSecretName: database-credentials
  externalAddress: *****
  k0sConfig:
    apiVersion: k0s.k0sproject.io/v1beta1
    kind: ClusterConfig
    metadata:
      name: k0s
    spec:
      network:
        provider: custom