search

seal-io / walrus

Walrus is an open-source application management platform based on IaC tools including OpenTofu, Terraform and others. It helps platform engineers build golden paths for developers and empowers developers with self-service capabilities.
Apache License 2.0
427 stars 48 forks source link

Fail to connect to the resource terminal (status code 1011) and sometimes panic #1257

Closed Finenyaco closed 1 year ago

Finenyaco commented 1 year ago

Environment

Steps to reproduce

  1. Add a template with source https://github.com/walrus-tutorials/helm-chart?ref=v0.0.1.
  2. Use the template to create a service. Fill in the chart_url https://github.com/walrus-tutorials/assets/releases/download/microservices-demo/sock-shop-0.3.0.tgz.
  3. Go to the service details page. Connect to the terminal of queue-master.

Result

  1. Failed to connect to the terminal. Check the logs, it returned status code 1011 instead of 1003.
image
  1. And sometimes fail to connect to the terminal will cause panic. 
    
2023-09-06T09:44:35.283Z    E   api error processing bidirectional stream request: error streaming exec: command terminated with exit code 126
github.com/seal-io/walrus/pkg/apis/runtime.doBidiStreamRequest
github.com/seal-io/walrus/pkg/apis/runtime/router_stream.go:239
github.com/seal-io/walrus/pkg/apis/runtime.(*Router).Routes.func1
github.com/seal-io/walrus/pkg/apis/runtime/router_route.go:139
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/auths.Account.Filter
github.com/seal-io/walrus/pkg/auths/account.go:56
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.RequestCounting.func3
github.com/seal-io/walrus/pkg/apis/runtime/middleware_flowcontrol.go:58
github.com/seal-io/walrus/pkg/apis/runtime.Per.func1
github.com/seal-io/walrus/pkg/apis/runtime/middleware_filter.go:84
github.com/seal-io/walrus/pkg/apis/runtime.If.func1
github.com/seal-io/walrus/pkg/apis/runtime/middleware_filter.go:58
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.RequestThrottling.func2
github.com/seal-io/walrus/pkg/apis/runtime/middleware_flowcontrol.go:86
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.I18n.func1
github.com/seal-io/walrus/pkg/apis/runtime/middleware_i18n.go:30
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.erroring
github.com/seal-io/walrus/pkg/apis/runtime/middleware_error.go:20
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.recovering
github.com/seal-io/walrus/pkg/apis/runtime/middleware_recovery.go:39
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.observing
github.com/seal-io/walrus/pkg/apis/runtime/middleware_observation.go:89
github.com/gin-gonic/gin.(*Context).Next
github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
github.com/gin-gonic/gin@v1.9.1/gin.go:620
github.com/gin-gonic/gin.(*Engine).ServeHTTP
github.com/gin-gonic/gin@v1.9.1/gin.go:576
github.com/seal-io/walrus/pkg/apis/runtime.(*Router).ServeHTTP
github.com/seal-io/walrus/pkg/apis/runtime/router.go:191
net/http.serverHandler.ServeHTTP
net/http/server.go:2947
net/http.(*conn).serve
net/http/server.go:1991
2023-09-06T09:44:35.283Z    D   api 1011 | HTTP/1.1 |       27 B |        0 B |   37.799412ms |   192.168.229.1 | BS      /v1/projects/477058412413248514/environments/477060886700286978/services/477078038316608258/resources/477078254054828802/exec?key=default-test%2Fqueue-master-6f5888bcd5-qj498%2Frun%2Fqueue-master&shell=bash
2023-09-06T09:44:35.283Z    E   Observed a panic: "send on closed channel" (send on closed channel)
goroutine 2971 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x46ed460?, 0x5f1cc50})
k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:75 +0x84
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x402588faf8?})
k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:49 +0x80
panic({0x46ed460, 0x5f1cc50})
runtime/panic.go:884 +0x20c
github.com/seal-io/walrus/pkg/apis/serviceresource.termStream.Read({{0x5f62738, 0x4000194000}, 0x40255d1690, {0x4025261440, 0x4025249020, {0x5f62770, 0x4025274480}, 0x40252724e0, 0x4025270c60, 0x4025261438, ...}, ...}, ...)
github.com/seal-io/walrus/pkg/apis/serviceresource/helper.go:61 +0x1b8
k8s.io/client-go/tools/remotecommand.readerWrapper.Read(...)
k8s.io/client-go@v0.26.2/tools/remotecommand/reader.go:40
io.copyBuffer({0xffff6ce30a00, 0x402583b360}, {0x5f2e1c0, 0x40248f9c00}, {0x0, 0x0, 0x0})
io/io.go:427 +0x168
io.Copy(...)
io/io.go:386
k8s.io/client-go/tools/remotecommand.(*streamProtocolV2).copyStdin.func1()
k8s.io/client-go@v0.26.2/tools/remotecommand/v2.go:103 +0xec
created by k8s.io/client-go/tools/remotecommand.(*streamProtocolV2).copyStdin
k8s.io/client-go@v0.26.2/tools/remotecommand/v2.go:95 +0x8c

github.com/seal-io/walrus/utils/log.DelegatedLogger.ErrorS github.com/seal-io/walrus/utils@v0.0.0-00010101000000-000000000000/log/delegate.go:208 github.com/seal-io/walrus/utils/log.logrSinker.Error github.com/seal-io/walrus/utils@v0.0.0-00010101000000-000000000000/log/logr.go:30 github.com/go-logr/logr.Logger.Error github.com/go-logr/logr@v1.2.4/logr.go:299 k8s.io/klog/v2.(loggingT).output k8s.io/klog/v2@v2.100.1/klog.go:879 k8s.io/klog/v2.(loggingT).printfDepth k8s.io/klog/v2@v2.100.1/klog.go:737 k8s.io/klog/v2.(loggingT).printf k8s.io/klog/v2@v2.100.1/klog.go:718 k8s.io/klog/v2.Errorf k8s.io/klog/v2@v2.100.1/klog.go:1578 k8s.io/apimachinery/pkg/util/runtime.logPanic k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:79 k8s.io/apimachinery/pkg/util/runtime.HandleCrash k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:49 runtime.gopanic runtime/panic.go:884 runtime.chansend runtime/chan.go:206 runtime.chansend1 runtime/chan.go:145 github.com/seal-io/walrus/pkg/apis/serviceresource.termStream.Read github.com/seal-io/walrus/pkg/apis/serviceresource/helper.go:61 k8s.io/client-go/tools/remotecommand.readerWrapper.Read k8s.io/client-go@v0.26.2/tools/remotecommand/reader.go:40 io.copyBuffer io/io.go:427 io.Copy io/io.go:386 k8s.io/client-go/tools/remotecommand.(streamProtocolV2).copyStdin.func1 k8s.io/client-go@v0.26.2/tools/remotecommand/v2.go:103 panic: send on closed channel [recovered] panic: send on closed channel

goroutine 2971 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x402588faf8?}) k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:56 +0xec panic({0x46ed460, 0x5f1cc50}) runtime/panic.go:884 +0x20c github.com/seal-io/walrus/pkg/apis/serviceresource.termStream.Read({{0x5f62738, 0x4000194000}, 0x40255d1690, {0x4025261440, 0x4025249020, {0x5f62770, 0x4025274480}, 0x40252724e0, 0x4025270c60, 0x4025261438, ...}, ...}, ...) github.com/seal-io/walrus/pkg/apis/serviceresource/helper.go:61 +0x1b8 k8s.io/client-go/tools/remotecommand.readerWrapper.Read(...) k8s.io/client-go@v0.26.2/tools/remotecommand/reader.go:40 io.copyBuffer({0xffff6ce30a00, 0x402583b360}, {0x5f2e1c0, 0x40248f9c00}, {0x0, 0x0, 0x0}) io/io.go:427 +0x168 io.Copy(...) io/io.go:386 k8s.io/client-go/tools/remotecommand.(streamProtocolV2).copyStdin.func1() k8s.io/client-go@v0.26.2/tools/remotecommand/v2.go:103 +0xec created by k8s.io/client-go/tools/remotecommand.(streamProtocolV2).copyStdin k8s.io/client-go@v0.26.2/tools/remotecommand/v2.go:95 +0x8c

Finenyaco commented 1 year ago

Environment

Test Result: no panic, but still unable to connect to the terminal.

image 
2023-09-11T05:23:41.962Z    E   api error processing bidirectional stream request: error streaming exec: command terminated with exit code 126
github.com/seal-io/walrus/pkg/apis/runtime.doBidiStreamRequest
    github.com/seal-io/walrus/pkg/apis/runtime/router_stream.go:239
github.com/seal-io/walrus/pkg/apis/runtime.(*Router).Routes.func1
    github.com/seal-io/walrus/pkg/apis/runtime/router_route.go:139
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/auths.Account.Filter
    github.com/seal-io/walrus/pkg/auths/account.go:56
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.RequestCounting.func3
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_flowcontrol.go:58
github.com/seal-io/walrus/pkg/apis/runtime.Per.func1
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_filter.go:84
github.com/seal-io/walrus/pkg/apis/runtime.If.func1
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_filter.go:58
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.RequestThrottling.func2
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_flowcontrol.go:86
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.I18n.func1
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_i18n.go:30
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.erroring
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_error.go:20
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.recovering
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_recovery.go:39
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/seal-io/walrus/pkg/apis/runtime.observing
    github.com/seal-io/walrus/pkg/apis/runtime/middleware_observation.go:89
github.com/gin-gonic/gin.(*Context).Next
    github.com/gin-gonic/gin@v1.9.1/context.go:174
github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
    github.com/gin-gonic/gin@v1.9.1/gin.go:620
github.com/gin-gonic/gin.(*Engine).ServeHTTP
    github.com/gin-gonic/gin@v1.9.1/gin.go:576
github.com/seal-io/walrus/pkg/apis/runtime.(*Router).ServeHTTP
    github.com/seal-io/walrus/pkg/apis/runtime/router.go:191
net/http.serverHandler.ServeHTTP
    net/http/server.go:2947
net/http.(*conn).serve
    net/http/server.go:1991
2023-09-11T05:23:41.962Z    E   write tcp 192.168.155.2:48614->198.19.249.2:26443: write: broken pipe
github.com/seal-io/walrus/utils/log.DelegatedLogger.ErrorS
    github.com/seal-io/walrus/utils@v0.0.0-00010101000000-000000000000/log/delegate.go:208
github.com/seal-io/walrus/utils/log.logrSinker.Error
    github.com/seal-io/walrus/utils@v0.0.0-00010101000000-000000000000/log/logr.go:67
github.com/go-logr/logr.Logger.Error
    github.com/go-logr/logr@v1.2.4/logr.go:299
k8s.io/klog/v2.(*loggingT).output
    k8s.io/klog/v2@v2.100.1/klog.go:879
k8s.io/klog/v2.(*loggingT).printDepth
    k8s.io/klog/v2@v2.100.1/klog.go:714
k8s.io/klog/v2.ErrorDepth
    k8s.io/klog/v2@v2.100.1/klog.go:1560
k8s.io/apimachinery/pkg/util/runtime.logError
    k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:115
k8s.io/apimachinery/pkg/util/runtime.HandleError
    k8s.io/apimachinery@v0.27.4/pkg/util/runtime/runtime.go:109
k8s.io/client-go/tools/remotecommand.(*streamProtocolV2).copyStdin.func1
    k8s.io/client-go@v0.26.2/tools/remotecommand/v2.go:104
2023-09-11T05:23:41.962Z    D   api 1011 | HTTP/1.1 |        1 B |        0 B |  235.210214ms |   192.168.155.1 | BS      /v1/projects/477934086535551746/environments/477937378493766402/services/477937955311229698/resources/477938055253170946/exec?key=default-orb-test%2Fqueue-master-6f5888bcd5-wxppn%2Frun%2Fqueue-master&shell=bash
thxCode commented 1 year ago

@Finenyaco please give the Kubernetes version here.

Finenyaco commented 1 year ago

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:20:54Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4+orb1", GitCommit:"40f54c2c6d383b1de832c2bc95f0966e047f7e7b", GitTreeState:"dirty", BuildDate:"2023-08-29T06:23:17Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"linux/arm64"}
Finenyaco commented 1 year ago

Environment

Test Result: pass

image