search

openyurtio / yurt-app-manager

The workload controller manager from NodePool level in OpenYurt cluster
Apache License 2.0
6 stars 1 forks source link

[BUG]yurt-app-manager invalid memory address or nil pointer dereference #48

Closed donychen1134 closed 2 years ago

donychen1134 commented 2 years ago

What happened: When I used yurt-app-daemon to deploy CoreDNS, I found that yurt-app-manager crashed.

+ kubectl --namespace=kube-system get pods
yurt-app-manager-76ccf4b55d-v88qs          0/1     CrashLoopBackOff   3409       12d
yurt-app-manager-76ccf4b55d-xlpx2          0/1     CrashLoopBackOff   3408       12d

E0412 05:56:13.005172       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 606 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1832520, 0x2704290)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/runtime/runtime.go:48 +0x89
panic(0x1832520, 0x2704290)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/openyurtio/yurt-app-manager/pkg/yurtappmanager/controller/yurtappdaemon.(*ReconcileYurtAppDaemon).cleanExpiredRevision(0xc00052adb0, 0xc000897040, 0xc000b31930, 0xc0006a9e00, 0x0, 0x0)
    /opt/src/pkg/yurtappmanager/controller/yurtappdaemon/revision.go:152 +0x4f
github.com/openyurtio/yurt-app-manager/pkg/yurtappmanager/controller/yurtappdaemon.(*ReconcileYurtAppDaemon).constructYurtAppDaemonRevisions(0xc00052adb0, 0xc000897040, 0xc000046040, 0xc000288720, 0xb, 0xc000288700, 0xf)
    /opt/src/pkg/yurtappmanager/controller/yurtappdaemon/revision.go:87 +0x17a
github.com/openyurtio/yurt-app-manager/pkg/yurtappmanager/controller/yurtappdaemon.(*ReconcileYurtAppDaemon).Reconcile(0xc00052adb0, 0x1c6c6a0, 0xc0008ef6b0, 0xc000288720, 0xb, 0xc000288700, 0xf, 0xc0008ef6b0, 0x40a1df, 0xc0006ae000, ...)
    /opt/src/pkg/yurtappmanager/controller/yurtappdaemon/yurtappdaemon_controller.go:146 +0x20e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000e2140, 0x1c6c5e0, 0xc0005411c0, 0x18a8ea0, 0xc0008cfe40)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263 +0x317
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000e2140, 0x1c6c5e0, 0xc0005411c0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1(0x1c6c5e0, 0xc0005411c0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000962750)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000b31f50, 0x1c2d1c0, 0xc0008ef5f0, 0xc000541101, 0xc0008162a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000962750, 0x3b9aca00, 0x0, 0x1, 0xc0008162a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1c6c5e0, 0xc0005411c0, 0xc0008cdf20, 0x3b9aca00, 0x0, 0x1)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1c6c5e0, 0xc0005411c0, 0xc0008cdf20, 0x3b9aca00)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:195 +0x4e7
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x168184f]
goroutine 606 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x1832520, 0x2704290)
    /usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/openyurtio/yurt-app-manager/pkg/yurtappmanager/controller/yurtappdaemon.(*ReconcileYurtAppDaemon).cleanExpiredRevision(0xc00052adb0, 0xc000897040, 0xc000b31930, 0xc0006a9e00, 0x0, 0x0)
    /opt/src/pkg/yurtappmanager/controller/yurtappdaemon/revision.go:152 +0x4f
github.com/openyurtio/yurt-app-manager/pkg/yurtappmanager/controller/yurtappdaemon.(*ReconcileYurtAppDaemon).constructYurtAppDaemonRevisions(0xc00052adb0, 0xc000897040, 0xc000046040, 0xc000288720, 0xb, 0xc000288700, 0xf)
    /opt/src/pkg/yurtappmanager/controller/yurtappdaemon/revision.go:87 +0x17a
github.com/openyurtio/yurt-app-manager/pkg/yurtappmanager/controller/yurtappdaemon.(*ReconcileYurtAppDaemon).Reconcile(0xc00052adb0, 0x1c6c6a0, 0xc0008ef6b0, 0xc000288720, 0xb, 0xc000288700, 0xf, 0xc0008ef6b0, 0x40a1df, 0xc0006ae000, ...)
    /opt/src/pkg/yurtappmanager/controller/yurtappdaemon/yurtappdaemon_controller.go:146 +0x20e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000e2140, 0x1c6c5e0, 0xc0005411c0, 0x18a8ea0, 0xc0008cfe40)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:263 +0x317
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000e2140, 0x1c6c5e0, 0xc0005411c0, 0x0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:235 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.1(0x1c6c5e0, 0xc0005411c0)
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:198 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000962750)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000b31f50, 0x1c2d1c0, 0xc0008ef5f0, 0xc000541101, 0xc0008162a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000962750, 0x3b9aca00, 0x0, 0x1, 0xc0008162a0)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1c6c5e0, 0xc0005411c0, 0xc0008cdf20, 0x3b9aca00, 0x0, 0x1)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1c6c5e0, 0xc0005411c0, 0xc0008cdf20, 0x3b9aca00)
    /go/pkg/mod/k8s.io/apimachinery@v0.19.7/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.7.0/pkg/internal/controller/controller.go:195 +0x4e7

I also found that when describing yurt-app-manager pod, there were some WARNING message.

Events:
  Type     Reason             Age                      From     Message
  ----     ------             ----                     ----     -------
  Warning  BackOff            9m27s (x79986 over 12d)  kubelet  Back-off restarting failed container
  Warning  MissingClusterDNS  4m23s (x86911 over 12d)  kubelet  pod: "yurt-app-manager-76ccf4b55d-v88qs_kube-system(4f0225b7-9bd5-4a25-95fa-19f14cfc620a)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.

What you expected to happen: yurt-app-daemon run normally.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

others Maybe the ud.Spec.RevisionHistoryLimit parameter is not initialized? https://github.com/openyurtio/yurt-app-manager/blob/master/pkg/yurtappmanager/controller/yurtappdaemon/revision.go#L152

The initialized position: https://github.com/openyurtio/yurt-app-manager/blob/master/pkg/yurtappmanager/controller/yurtappdaemon/yurtappdaemon_controller.go#L131

/kind bug

rambohe-ch commented 2 years ago

@donychen1134 Thanks for raising issue to report bug. would you be able to make a pull request to solve this bug?

donychen1134 commented 2 years ago

@rambohe-ch I'll have a try. /assign @donychen1134

donychen1134 commented 2 years ago

After reinstalling yurt-app-manager, this problem cannot be saw again. So I will close this issue now.

donychen1134 commented 2 years ago

/close

openyurt-bot commented 2 years ago

@donychen1134: Closing this issue.

In response to [this](https://github.com/openyurtio/yurt-app-manager/issues/48#issuecomment-1125631669): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.