Closed swoldanski closed 4 years ago
Describe the bug When you try to deploy service with --request-timeout-seconds flag rio-controller crashes and the system is not working anymore.
To Reproduce Steps to reproduce the behavior:
Expected behavior Service is created and specified timeout is set in the system. Rio does not crash and is working.
Kubernetes version & type (GKE, on-prem): kubectl version
kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:29Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.5", GitCommit:"20c265fef0741dd71a66480e35bd69f18351daea", GitTreeState:"clean", BuildDate:"2019-10-15T19:07:57Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}
Type: Rio version: rio info
rio info
Rio Version: v0.6.0 (c1bff129) Rio CLI Version: v0.6.0 (c1bff129) Cluster Domain: m2g2ms.on-rio.io Cluster Domain IPs: System Namespace: rio-system Wildcard certificates: true
Additional context rio system logs output:
rio system logs
rio-controller | time="2020-01-25T14:22:28Z" level=info msg="Starting rio-controller, version: v0.6.0, git commit: c1bff129" rio-controller | time="2020-01-25T14:22:28Z" level=info msg="Updating CRD services.rio.cattle.io" rio-controller | time="2020-01-25T14:22:28Z" level=info msg="Updating CRD stacks.rio.cattle.io" rio-controller | I0125 14:22:29.226694 1 leaderelection.go:241] attempting to acquire leader lease rio-system/rio... rio-controller | time="2020-01-25T14:22:29Z" level=info msg="listening at :443" rio-controller | I0125 14:22:29.245247 1 leaderelection.go:251] successfully acquired lease rio-system/rio rio-controller | time="2020-01-25T14:22:29Z" level=info msg="Starting /v1, Kind=ConfigMap controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting apps/v1, Kind=StatefulSet controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting tekton.dev/v1alpha1, Kind=TaskRun controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting apps/v1, Kind=Deployment controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting cert-manager.io/v1alpha2, Kind=Certificate controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting /v1, Kind=Service controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting /v1, Kind=Secret controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting admin.rio.cattle.io/v1, Kind=ClusterDomain controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting admin.rio.cattle.io/v1, Kind=PublicDomain controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting rio.cattle.io/v1, Kind=Router controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting rio.cattle.io/v1, Kind=Stack controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting rio.cattle.io/v1, Kind=Service controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting rio.cattle.io/v1, Kind=ExternalService controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting gloo.solo.io/v1, Kind=Settings controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting extensions/v1beta1, Kind=Ingress controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting gitwatcher.cattle.io/v1, Kind=GitCommit controller" rio-controller | time="2020-01-25T14:22:33Z" level=info msg="Starting /v1, Kind=Endpoints controller" rio-controller | E0125 14:27:28.623151 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) rio-controller | goroutine 885 [running]: rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1c18d00, 0x343d1b0) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3 rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82 rio-controller | panic(0x1c18d00, 0x343d1b0) rio-controller | /usr/local/go/src/runtime/panic.go:679 +0x1b2 rio-controller | github.com/rancher/rio/modules/gloo/pkg/vsfactory.(*VirtualServiceFactory).ForRevision(0xc0007875c0, 0xc001135680, 0xc0016e6f00, 0x1fddf80, 0xc000d5ab98, 0xc000d5aad0, 0xc000d5ac78) rio-controller | /go/src/github.com/rancher/rio/modules/gloo/pkg/vsfactory/revision.go:23 +0x5c1 rio-controller | github.com/rancher/rio/modules/gloo/controller/service.(*handler).generate(0xc000922f60, 0xc001135680, 0x0, 0x0, 0xc00275f4a0, 0xf, 0xc00275f47a, 0x2, 0x0, 0x0, ...) rio-controller | /go/src/github.com/rancher/rio/modules/gloo/controller/service/handler.go:45 +0x6f rio-controller | github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1.(*serviceGeneratingHandler).Handle(0xc0000cc900, 0xc001135680, 0x0, 0x0, 0xc00275f4a0, 0xf, 0xc00275f47a, 0x2, 0x0, 0x0, ...) rio-controller | /go/src/github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1/service.go:315 +0x90 rio-controller | github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1.(*serviceStatusHandler).sync(0xc0009231a0, 0xc000fd9600, 0x1f, 0xc001135200, 0xc00100dd90, 0x0, 0x0) rio-controller | /go/src/github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1/service.go:282 +0x11f rio-controller | github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1.FromServiceHandlerToHandler.func1(0xc000fd9600, 0x1f, 0x22077c0, 0xc001135200, 0x1, 0xc00100dd90, 0x0, 0x1) rio-controller | /go/src/github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1/service.go:101 +0x65 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Handlers).Handle(0xc000a74d80, 0xc000fd9600, 0x1f, 0x22077c0, 0xc001135200, 0x1c62ce0, 0x1ed60a0, 0xc000d9c900, 0x22077c0) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/handlers.go:27 +0x100 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).syncHandler(0xc0002c4d20, 0xc000fd9600, 0x1f, 0x138a881, 0xc000b5bf88) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:175 +0xdb rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).processSingleItem(0xc0002c4d20, 0x1b6a640, 0xc00100d710, 0x0, 0x0) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:156 +0xf0 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).processNextWorkItem(0xc0002c4d20, 0x225c3a225c6b6f01) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:133 +0x51 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).runWorker(0xc0002c4d20) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:122 +0x2b rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000761270) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000761270, 0x3b9aca00, 0x0, 0x687361682f6f6901, 0xc000127440) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8 rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000761270, 0x3b9aca00, 0xc000127440) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d rio-controller | created by github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).run rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:104 +0x178 rio-controller | panic: runtime error: invalid memory address or nil pointer dereference [recovered] rio-controller | panic: runtime error: invalid memory address or nil pointer dereference rio-controller | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x18c9101] rio-controller | rio-controller | goroutine 885 [running]: rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105 rio-controller | panic(0x1c18d00, 0x343d1b0) rio-controller | /usr/local/go/src/runtime/panic.go:679 +0x1b2 rio-controller | github.com/rancher/rio/modules/gloo/pkg/vsfactory.(*VirtualServiceFactory).ForRevision(0xc0007875c0, 0xc001135680, 0xc0016e6f00, 0x1fddf80, 0xc000d5ab98, 0xc000d5aad0, 0xc000d5ac78) rio-controller | /go/src/github.com/rancher/rio/modules/gloo/pkg/vsfactory/revision.go:23 +0x5c1 rio-controller | github.com/rancher/rio/modules/gloo/controller/service.(*handler).generate(0xc000922f60, 0xc001135680, 0x0, 0x0, 0xc00275f4a0, 0xf, 0xc00275f47a, 0x2, 0x0, 0x0, ...) rio-controller | /go/src/github.com/rancher/rio/modules/gloo/controller/service/handler.go:45 +0x6f rio-controller | github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1.(*serviceGeneratingHandler).Handle(0xc0000cc900, 0xc001135680, 0x0, 0x0, 0xc00275f4a0, 0xf, 0xc00275f47a, 0x2, 0x0, 0x0, ...) rio-controller | /go/src/github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1/service.go:315 +0x90 rio-controller | github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1.(*serviceStatusHandler).sync(0xc0009231a0, 0xc000fd9600, 0x1f, 0xc001135200, 0xc00100dd90, 0x0, 0x0) rio-controller | /go/src/github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1/service.go:282 +0x11f rio-controller | github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1.FromServiceHandlerToHandler.func1(0xc000fd9600, 0x1f, 0x22077c0, 0xc001135200, 0x1, 0xc00100dd90, 0x0, 0x1) rio-controller | /go/src/github.com/rancher/rio/pkg/generated/controllers/rio.cattle.io/v1/service.go:101 +0x65 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Handlers).Handle(0xc000a74d80, 0xc000fd9600, 0x1f, 0x22077c0, 0xc001135200, 0x1c62ce0, 0x1ed60a0, 0xc000d9c900, 0x22077c0) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/handlers.go:27 +0x100 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).syncHandler(0xc0002c4d20, 0xc000fd9600, 0x1f, 0x138a881, 0xc000b5bf88) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:175 +0xdb rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).processSingleItem(0xc0002c4d20, 0x1b6a640, 0xc00100d710, 0x0, 0x0) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:156 +0xf0 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).processNextWorkItem(0xc0002c4d20, 0x225c3a225c6b6f01) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:133 +0x51 rio-controller | github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).runWorker(0xc0002c4d20) rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:122 +0x2b rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000761270) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000761270, 0x3b9aca00, 0x0, 0x687361682f6f6901, 0xc000127440) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8 rio-controller | github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000761270, 0x3b9aca00, 0xc000127440) rio-controller | /go/src/github.com/rancher/rio/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d rio-controller | created by github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic.(*Controller).run rio-controller | /go/src/github.com/rancher/rio/vendor/github.com/rancher/wrangler/pkg/generic/controller.go:104 +0x178
Ok, it looks that this flag is working as expected in Rio v0.7.0. The simplest path to solve this issue is upgrading your Rio to v0.7.0
Describe the bug When you try to deploy service with --request-timeout-seconds flag rio-controller crashes and the system is not working anymore.
To Reproduce Steps to reproduce the behavior:
Expected behavior Service is created and specified timeout is set in the system. Rio does not crash and is working.
Kubernetes version & type (GKE, on-prem):
kubectl version
Type: Rio version:
rio info
Additional context
rio system logs
output: